Big Data Pitfalls

Avoid Simpson’s paradox:
This paradox refers to a phenomena where the association between a pair of variables (X; Y) reverses sign upon conditioning of a third variable, Z regardless of the value taken by Z. If we partition the data into subpopulations, each representing a specic value of the third variable, the phenomena appears as a sign reversal between the associations measured in the disaggregated subpopulations relative to the aggregated data, which describes the population as a whole.

Right ML algorithms usage: use the right approach for machine learning algorithms, find the appropriate algorithm for your specific problems. Ex. If you need a numeric prediction quickly, use decision trees or logistic regression.

Keep in mind the Prisoner’s Dilemma: like in “cigarette manufacturers endorsed the making of laws banning cigarette advertising, understanding that this would reduce ad costs for parties and increase profits across the industry”, so it is with the business strategy and down to big data processing.

Consider Gödel’s Theorem: any system of computation you can construct (numbers theory etc.) that it is true, it cannot be ultimately proved from the rules within that computational construct. The system in a way transcends itself. Thus the way to the strong AI for example.

Keep in mind the exponentially powerful quantum computers of the future. For example build different, resistant cryptographic algorithms against the qubits future powers.

AI Weak and Strong

Capabilities currently classified as weak AI include successfully understanding human speech, competing at a high level in strategic game systems (such as chess and Go), self-driving cars, intelligent routing in content delivery networks, and interpreting complex data. Done through mainly calculus involving prediction problems, extrapolation based on feeding quality data.

The central problems (or goals) of AI research include reasoning, knowledge, planning, learning, natural language processing (communication), perception and the ability to move and manipulate objects.

General intelligence is among the field’s long-term goals. Approaches include statistical methods, computational intelligence, and traditional symbolic AI. Many tools are used in AI, including versions of search and mathematical optimization, logic, methods based on probability and economics. The AI field draws upon computer science, mathematics, psychology, linguistics, philosophy, neuroscience and artificial psychology.

From a deep learning perspective, even neuroevolution approaches has its limits for a strong AI, no matter how much the population of networks grow to attain the objective the acknowledgment of the solution has to have an external “decidant.”

“That is, evolution in this case is not just deciding the architecture and weights, but also the rules that guide how and when particular weights change.” from Neuroevolution: A different kind of deep learning by O.Stanley

In the context of strong AI, I really think that we need a kind of “attractors” framework as a “loosely” decidant role (aka. “conscience”).

“Strong AI is related to the field of consciousness, sentience, mind. We are on the way to general intelligence with every AI field evolving (from natural language processing to cognitive computing, social intelligence, planning, learning, perception etc.) The rules that govern the universe are already there, our mind works under the patterns within the universes pattern. To realize the strong AI we need to hook the machine into this flow of our as-such universe of small-world networks*. Consciousness is not programmable, it is a result of the sensory, when hooked to the flow, it can be seen as a result of the “strange attractors” created into the flow. A strong AI will exist when all the sensory inputs will have a proper cognitive processing, conscience will arise de facto from the space where the inputs exists but under a proactive threshold. The triggering occurs under strange attractors influenza, thus the conscience manifests itself.” (Definition of Strong AI by Essential-Works, C.Stefan 04.2017).

* See: Watts and Strogatz model & Barabási–Albert model.
* Restricted Boltzmann Machine and Self-Organization systems#.
* Active inference#.