Maybe we can look for historical examples of how legislation had a positive impact on a widespread harmful habit.
Early on, the U.S. Congress adopted the Federal Cigarette Labeling and Advertising Act of 1965 and the Public Health Cigarette Smoking Act of 1969. These laws—
* Required a health warning on cigarette packages
* Banned cigarette advertising in the broadcasting media
* Called for an annual report on the health consequences of smoking
I'm as impressed as anyone with GPT-3 samples, but you're sort of ignoring the symbol grounding elephant in the room regarding language models (https://openreview.net/pdf?id=GKTvAcb12b).
Language models are not grounded learners. The language produced does not really correspond meaningfully to our world except in superficial (albeit complex) ways.
Do you have thoughts on how to move forward on this problem? Maybe ask GPT-3 and see what it thinks :P
The problem, if I understand correctly, is that we're feeding enormous amounts of text to language models hoping that they might contain, hidden in their patterns, enough information about the real world to allow prodigiously complex NNs to extract it and create their own representation of reality.
And while this is possible, it feels there should be more effective ways to impart a knowledge of reality- if only we had huge databases of usable data to feed to these NNs instead of dumps of text. At the moment it feels like we're trying to teach advanced physics to a subject with no previous knowledge of physics or math by just feeding it with everything on arXiv and physics textbooks in random order. What you get is someone who can produce text that mimics the superficial style of scientific articles, but with an extremely confused understanding of the subject, if any at all.
I am happy to take them at their word that their theory about symbol grounding proves that no LM will ever be able to solve "Three plus five equals" (appendix B); and thus, by modus tollens, GPT-3's ability to (already) solve "Three plus five equals" means their theory is wrong and I need not consider it any further.
Symbol grounding is as much a problem in AI as whether or not our use of language is meaningful. Does our language encode particular models of the world? Yes? Good. Then AI models also encode models of the world.
They have published multiple major TTS papers since the original WaveNet paper in 2016. Including the recent Tacotron results with impressive style control.
This is called federated learning[0] at least by Google. I don't know whether they've added this to more products or whether it works well. It would be interesting to see this done in open source.
They're most likely referring to adversarial attacks where degenerate inputs are constructed that could cause AlphaGo Zero to perform sub-optimally or catastrophically fail (see OpenAI Research [0]). This is distinct from generative adversarial networks (GANs) or adversarial self-play (which I guess AlphaGo Zero is an example of).
It was trained with self-play and evaluated on AlphaGo Lee (the version that beat Lee Sedol) and AlphaGo Master (the version that beat Ke Jie and I believe a number of online matches against top-ranked professional players).
>No one is ever going to use word vectors to figure out what genders are capable of what jobs.
How can you possibly make this claim?
Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).
It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.
Have you seen the results from Dynamic Memory Networks? [0]
The relevant example from the paper:
I: Jane went to the hallway.
I: Mary walked to the bathroom.
I: Sandra went to the garden.
I: Daniel went back to the garden.
I: Sandra took the milk there.
Q: Where is the milk?
A: garden
Obviously just a toy task, but as you said, progress is rapid!
DeepMind's main intellectual property, the reason why it was acquired, was DeepMind's reinforcement algorithm. The present paper, like the work on AlphaGo, is about improving this reinforcement algorithm.
I don't believe that's the case. The Atari results were achieved via Q-Learning with neural networks (DQN stands for Deep Q-Networks), both of which have been around for decades.
The basic algorithm is something you can implement yourself fairly easily, but (as with most things involving neural nets) getting it to really perform takes a bit of expertise.
So, DeepMind's greatest asset is they employ some really excellent people and have a substantial head start in terms of actually implementing AIs and getting them to work.
