I have conducted penetration tests where we redefined network truth for a client. This is a high offense energy state where the adversary is able to alter the fundamental assumptions of a target network. This process breaks security assumptions and opens up many attack avenues.
Network truth is things like firewall positioning in IP flows or that names resolve to things you expect them to. Secuirty architects laid out controls based on the truth of the network design and expected attack vectors. If the network changes due to something in the world or a nefarious actor then it will suddenly introduce more risk.
Broaden it out to the world and think about how you learn truth. Where are your assumptions about what can and cannot be trusted? Just a few years ago various groups were up in arms around the gate keeping of search engines like Google to show certain things but not others. Now we have more and more people getting their information from LLMs. LLMs trained by people they didn’t know on data they have little visibility into and accessed through an application they can’t trust.
Will this scenario lead to truth? Possibly, but it isn’t designed to end up there. Is it more likely to end up as propaganda and censorship? What if not even so overt, would you know if an LLM response was nudging you in a certain direction? Routing you around a mental firewall and changing some assumptions?
The challenge with most psychologial risks is the difficulty of picking them out in any one instance. You need aggregate data points with vast visibility that most simply don’t have.
The solution is a new truth seeking llm designed with the assumptions of an adversarial world. A system that can evaluate lots of information and determine what is most likely to be true. Such a system will need to be trained on adversarial techniques as well as classical thought and reasoning for how to determine something is credible.
Just like in penetration testing, you don’t know the whole environment, just pieces of data you collect and then must make inferences based on likelihood. A port being listed as ‘filtered’ most likely means a firewall dropping the packet but all you really know is that a packet didn’t make it back for which there are many other, less likely, candidates.
Fortunately the nature of core machine learning is probabilities and this kind of analysis is where the technology shines. Foundation LLMs will be able to study patterns of what predicts well and use this to navigate inherently untrusted environments like the world.
Truth discerning ai will be a powerful companion and guardian of humanity in the coming golden age.
Leave a Reply