The abyss is a sort of post-apocalyptic land located beyond and underneath the visible surface of Teyvat. It came about after the fall of the Eclipse Dynasty of Khaenri'ah (ouch) 500 years ago and is currently inhabited by the Abyss Order. There are likely more inhabitants of the Abyss, we simply haven't met them yet! It is home to an unworldly power that opposes that of Celestia, which allows them to corrupt beings such as Dvalin and Durin.
The Abyss is home to the Abyss Order, a group led by your twin that persistently seeks out the ruins of Khaenri'ah.
"Only the Abyss Order has consistently sought out the remnants of Khaenri'ah. Despite being far from human, they seek out this lost human civilization quite persistently." — Dainsleif, A Herald Without Adherents
When Khaenri'ah was destroyed, a great sinner created endless monsters with dark, alien blood flowing through their veins. They rampaged across the land, destroying all in their paths. — Chunk of Aerosiderite
Abyss Mages — These creatures are the lowest of the Abyss Order hierarchy. They are able to manipulate the elements through the use of ley lines (which are dropped when defeated) and communicate with hilichurls. So far, they have powers over Pyro, Cryo, and Hydro elements. They are remnants of Khaenri'an citizens, cursed into non-human monsters.
Abyss Heralds — Upgraded soldiers of the Abyss Order. They are able to communicate with humans and pose a greater threat than Abyss Mages. They are also able to summon portals and easily travel between the surface world and Abyss as seen in We Will be Reunited. Currently, they are only seen with a Hydro element.
They are able to undergo an unknown transformation using a power called Abyss Gospel. This item is yet to be seen in-game as of the second Dainsleif quest. As Abyss Heralds don't seem to possess visions, it's likely that they also gain elemental powers through the manipulation of ley lines.
Skirk — ****A mysterious lone swordswoman who lives in the Abyss. Not much is known about her, apart from the fact that she saved and trained Childe during his time in the Abyss. She seems to be very powerful, but remains unmentioned apart from Childe's backstory.
Irminsul Tree and Ley Lines — The powerful Irminsul tree is said to grow up from the bottom of the world and bloom on the surface. This tree gives birth to the ley lines seen across Teyvat. The Abyss Mages draw their power from this tree, which allows them to manipulate the elements. Ley lines also give root to the Moonlit Tree in Nantianmen.
Abyss Twin — Your twin. Recently rose as leader of the Abyss Order, for somewhat unknown circumstances. They're referred to as prince/princess by the members of the Abyss Order, and won't leave Teyvat until the Abyss Order has accomplished its goals—no matter what you say.