I've juts collected all the evidence we have about "why Genjutsu works without chakra".

This dumb argument has to stop. It was never stated anywhere that Genjutsu works by controlling the chakra flow, what is said is that Genjutsu affects the senses to control the chakra flow not the other away around. You people are making as much sense as someone who interprets "A water-pump is something that works on electricity to control water flow" as " Water-pump controls water to work on electricity".

The other silly argument people tend to use to claim that Genjutsu works by controlling chakra is the fact that using a burst of chakra can break Genjutsu, but we have seen that certain types of Genjutsu can also be broken through pain so does this mean that Genjutsu works by controlling pain as well, as opposed to it being an effect of Genjutsu ? Obviously not...

Furthermore what those people like to conveniently forget is the fact that chakra was given to humans by the sage of six paths, the man who confessed that he didn't like the way his mother was using Genjutsu to control people and searched for ways to dispel it, so how very surprising indeed that the chakra he gave to humanity can free them from Genjutsu...

Also, you speak of the chakra pathway system but it seems that you don't know much about it. Just because Shinobi can create chakra doesn't mean that everyone and their grandmother in Naruto go around carrying a mass of chakra inside of them. It has been explained since part 1 that chakra is something Ninjas create consciously when they need to, not at all times.

^ As the above chart shows and explains, chakra is something that is consciously created for the use of Ninjutus or Genjutsu. Various showings through the series confirm this. just to use a couple :

Naruto building up chakra to use the Summoning jutsu, but getting that chakra absorbed by Kisame, conclusion : "I can't feel any chakra".

Gallery

Tobirama discussing the village's future with Hashirama, but not realizing that Madara was spying on them, reason : "I'm not kneading any chakra"

A more practical display of this comes from the Byakugan vision. For instance.

Neji's body while not using chakra and when using chakra :

Kidomaru's chakra network when preparing to use a jutsu :

^ Notice how chakra only flows through the parts related to the jutsu.

And yet you people insist that Genjutsu can only work on chakra, even though we have seen Kabuto and Itachi Genjutsu fodder villagers... so tell me why would the villagers and some random woman, who for all we know were not even trained to create chakra in the first place, be creating chakra and having it flow through their heads for no apparent reason whatsoever, just so Itachi and Kabuto can use it to genjutsu them ? The answer is that they weren't, and Genjutsu does not need chakra to work.

• 1: Infinite Tsukuyomi is a Genjutsu. Kaguya put the world in a massive Genjutsu, before chakra creation. Saying "it was the strongest Genjutsu so nope. Still Genjutsu can't affect people without chakra 'cause only Inf. Tsukuyomi can" it's very very hilarious. That's a Genjutsu, end of story.
  

• 2: Another example is in the Itachi's light novel. On, i believe, page 30, Itachi was talking to Shisui, and Shisui explains that occular Genjutsu and Genjutsu like it can affect anyone because it doesn't disrupt chakra like other basic Genjutsu.
  He explains that Genjutsu affect the nervous system and mind itself and fills the gap by pouring their own chakra as a medium.
  

  If jiraiya's quote isn't enough to convince you, another quote by another character saying the exact same thing should work:

• 3: Boruto The Movie: scientists created a ninja tool that allows people without chakra to use jutsu. They wouldn't need to create a tool like that if everyone had chakra, which means not everyone has chakra. Which means yes, putting villagers under a Genjutsu means you didn't manipulate the chakra in their cranial nerves, something that, again, is only mentioned in a wiki, and again only proves that Genjutsu targets the senses.

So now that that is settled, we can all agree that:

• There is no scan in the whole Naruto manga stating that Genjutsu requires a target to have chakra, whereas there is more than one scan explicitly stating that Genjutsu targets the senses.
• Not everyone in the Narutoverse has chakra constantly in their body, they have to first be trained to make it and then actively make it, which is proven numerous times in the manga.
• Kaguya's usage of IT isn't an exception to the rule because there wasn't a rule in the first place because there are literally no scans stating such a rule in the entirety of the manga. For it to be the exception, there would have to be an example of a Genjutsu failing to work because a target didn't have chakra.

Simple Example:

Chakra must be created first, as already proved. And in order to create chakra training is required (ex. Shinobi: Genin, Chunin, Jonin etc).

• After ^ I'll show you Kabuto performing a Genjutsu, "Temple of Nirvana", and put every normal person into an induced instinctive-sleep (Genin included, like Naruto).

• This is a normal girl that was mind controlled by Itachi with a Sharingan Genjutsu.

• In the Boruto manga Toono Katasuke was put under a Genjutsu, mind control, by Kashin Koji.

Special thanks to @mudamudamuda and @thevivas