Knowledge market is a distributed social mechanism that helps people to identify and satisfy their demand for knowledge. It can facilitate locating existing knowledge resources similarly to what search engines do (the name social search refers to this). It can also stimulate creation of new knowledge resources to satisfy the demand (something that search engines can’t do). The goal of this post is to compare several free knowledge markets created by 3form, Naver, Yahoo, Mail.ru, and Google to identify their common elements and differences. All these sites organize collaborative problem solving activity of a large number of participants providing means and incentives to contribute participants’ intelligent abilities to the distributed problem solving process. MIT Tech Review published an attempt to make a comparison of Q&A sites by Wade Rush. Unfortunately, that comparison was of low quality and too superficial to be useful (see the readers comments). Here is my attempt at such a comparison. Its focus is on free knowledge markets, i.e. those that don’t charge fees for participation and allow participants to build on top of the knowledge resources contributed by others.
Background
Free Knowledge Exchange project was launched in summer 1998 in Russia and its international version became available at 3form.com in February 1999. The project allows any participant to submit problems and brings those problems to the attention of other people (3form community) to collect hints and solutions. The credit assignment system of the website tracks contribution of each individual participant to solving problems. It rewards the actions of the participants as well as the quality of contributed content. In exchange for the contribution to solving problems of others, the website returns to the participant the proportional share of the collective attention directed towards solving the participant’s own problems. 3form uses the method known as human-based genetic algorithm (published in 2000). Naver Knowledge iN is a Korean free knowledge market service opened in 2002 by NHN Corp. The site is based on the same idea as 3form, though implements it somewhat differently. This service made Naver the biggest internet destination in Korea and was a major factor allowing Naver to beat Google and Yahoo in the Korean search market. It took a couple of years for Yahoo and Google to learn their Korean lesson. Yahoo launched Yahoo Answers in December 2005 that is now the biggest free knowledge market worldwide. Mali.ru is a Russan knowledge market inspired by Yahoo Answers and currently the biggest similar service in Russia. Google Q&A is the newest service being tested in Russia and China by Google. Google’s service is likely to be inspired by prior work, though I am not aware of Google acknowledging this.
Prior to 3form, two ways of collective problem solving on the internet were available. On one hand, there were free knowledge sharing forums such as Usenet and IRC, where users could ask technical questions and get help from volunteer experts whose participation was neither accounted nor rewarded in any way. On the other hand, there were expert advice services designed around fee-based Q&A model where questions are answered for a fee by a limited number of paid experts. Experts Exchange (EE) made a step towards becoming a free service by allowing anyone to answer questions. It introduced “answer points” to identify experts among its volunteer answerers. The answer points were awarded based on user evaluation of the answers: the author of the problem could allocate the total amount of answer points among all people who contributed useful ideas toward solution. Despite of beign innovative at the expert side, the service remained fee-based on the user side (even though users were getting some credit in “question points” independent of their contribution, allowing them to ask a limited number of questions). In other words, while the question points had monetary value, answer points had no such value (in particular they couldn’t be counted towards “question points”). Once the limited amount of question points was exhaused, users had to start buying question points to continue using the system.
Korean Naver played a key role in popularizing the concept of knowledge market. Naver, however, was not the first knowledge market in Korea. DBDiC offered analogous service as early as Oct 2000. DBDiC presumably developed its technique independently from 3form, but shares similar architecture, including general structure and credit assignment system. There are two key differences of DBDiC technique from the one of 3form: (1) the identity of the author of the solution biases evaluation of the solution, i.e. high status of the author can lead to accepting the inferior answer as the best despite the presence of the better answer contributed by a person with lower status, (2) answers that are positioned earlier in the list are more likely to be read, chosen, and evaluated, i.e. a great answer later in the list can easily be overlooked. These differences resulted in subjective and position specific biases in solution evaluation system (see also my previous post .Bugs of collective intelligence: why the best ideas aren’t selected?). I could speculate that if DBDiC designers were more familiar with 3form service, they could have avoided those undesirable biases that later propagated into every knowledge market platform created subsequently.
Free knowledge markets are currently abundant with numerous implementations. Wade Rush lists six recently created services. ReadWriteWeb post lists 29 knowledge market services. Neither of these lists is complete and it may not be feasible to create a complete list as new similar services appear almost every day. However, most new services are similar to one of those reviewed here and are likely to be inspired by them.
Incentive systems
Knowledge markets differ from knowledge sharing websites by implementing knoweldge evaluation and incentive systems that encourage participants to help each other. The incentive system of a typical free knowledge market is based on rewarding actions of its participants as well as rewarding quality of their contribution. Measure of quality is normally based on user evaluation. An alternative to this would be computational evaluation, e.g. one based on frequency of occurrence of different answers collected independently (see Luis von Ahn work that explores this model in specialized applications like image labeling). In the case of a typical knowledge market frequency counting is problematic due to the bigger search space.
The action rewards encourage particular actions reinforcing participant behavior that is beneficial for the system as a whole, for example, it can be as simple as visiting the website. The following table summarizes action rewards offered by different knowledge markets:
| Action | 3form | Naver | Yahoo | Mail.ru | |
|---|---|---|---|---|---|
| Join | 1 | 100 | 100 | 100 | |
| Visit | 3 | 1 | 1 | 5 | |
| Submit question anon | 0 | -20 | N/A | N/A | N/A |
| Submit question pseu | N/A | 1 | -5 | -5 | -B |
| Submit expert question | N/A | -50*E | N/A | N/A | N/A |
| Submit answer | 0.01 | 2 | 2 | f(K) | 2 |
| Select best answer to your question | 3 | 3 | |||
| Evaluate answer | 1 | 1 | [S>=250] | 1 | |
| Evaluate question | 1 | 1 |
In this table, S refers to the current score of the participant. For example, Mail.ru will not reward new participants for provided evaluations until they get score of 250 points. This seems to be an effective way to guard the system from abuse. With this system in place, it becomes hard for someone to manipulate the values of the answers in Mail.ru by creating multiple accounts and submitting votes from them. It is no longer enough to create multiple accounts, it is also required to earn 250 points of credit for each, which will protect the system from bots better than any CAPTCHA would do. Mail.ru also rewards non-peer reviewed answers differently, depending on the prior performance of their author. For this purpose Mail.ru designers inroduced “Energy Conversion Coefficient” which is simply the share of the best answers to the total number of answers that the person generated. This seems to be a good incentive to provide quality answers and so far this is a unique feature of Mail.ru answers. B stands for bonus, a number from 1 to 100 that is set by the author of a question. The bonus can reflect the difficulty or importance of a problem for its author and supposedly high bonus will motivate people to pay more attention to the question. E is the number of experts in Naver’s expert question, can be 1 or 2 (not available in other services).
The quality evaluation rewards are summarized below:
| Evaluation | 3form | Naver | Yahoo | Mail.ru | |
|---|---|---|---|---|---|
| Question reward | 0.02*R | 1 | |||
| Answer reward | R*log(E) | ||||
| Best answer | 0.03*A | 10 | 10 | 10 | B |
Features
Common features comparison:
| Feature | 3form | Naver | Yahoo | Mail.ru | |
|---|---|---|---|---|---|
| Question bookmarking | Y | Y | Y | Y | Y |
| Question evaluation | Y | Y | N | N | |
| Answering your own question | Y | N | N | Y | Y |
| Submitting multiple answers | Y | N | N | N | Y |
| Search questions asked by others | N | Y | Y | Y | Y |
| Search returns how many answers? | N/A | 1 | All | All | All |
| How many answers can be selected as best? | 1 | 2 | 1 | 1 | 1 |
| Question open (days) | until removed | 2-15, default 5 | 4 answ/1 vote | 5 | 5 |
| Innovation/Selection | conc | seq | seq | seq | seq |
| Social networking | N | Y | Y | Y | Y |
Yahoo Answers explicitly forbids you to answer your own quesiton: “You can’t answer your own question.” It forbids to submit another answer if one is already submitted. 3form and Google allow this. In fact, people can use Google system as a discussion forum and post comments and additions to the previous answers (this requires to keep answers in the order they were received, i.e. creates temporal selection bias towards earlier answer).
Unique features
- 3form doesn’t have subjective and temporal biases of evaluation present in other systems. This improves the chances that the best answers will be selected. The amount of attention each problem receives is proportional to the amount of attention its author (and other people interested in this problem) paid to solving problems of others.
- Naver’s registration involves cell phone verification. If you want to register, you have to provide you cell phone to Naver and then input a verification code sent to your cell phone. This makes sockpuppetry (a practice of establishing several accounts to influence the system) much harder than simple email verification used in other services. Owners of several cell phone numbers still can have multiple accounts. Another specific thing about Naver is that most of the questions ask for information rather than knowledge. Maybe it partly explains that our question about the first knowledge network was too unusual for Naver, so it didn’t receive any answers
- Mail.ru uses cell phone messaging to auction a limited number of featured questions. The users compete for a limited space by sending multiple IM messaged to Mail.ru number from their cell phones (and paying IM fees). Questions of people who sent the largest number of messages are featured at the front page. In addition to this, Mail.ru has a button, “Send thanks to the answerer.” If this button is pressed the message pops up suggesting to send a text message from a cell phone to Mail.ru number, each thank you costs $1 at Mail.ru. Mail.ru allows to search questions, however this search couldn’t find anything when I entered keywords for my question. It assume it takes time for new questions to become searchable
- Google has extended question exposure statistic: number of times the question was shown
A small empirical test
The purpose of this test was not determining “the best” Q&A site as in the MIT Tech review comparison. My purpose is to give some information on what kind of results you could expect from using free knowledge market services. I needed this test mainly to see how the websites work and plan to do more extensive testing later.
Recent article at inc.com How to kill a great idea claims that “Jonathan Abrams created the first online social network” right from the beginning. Wikipedia suggests that at least Classmates.com and SixDegrees.com were created earlier than Friendster and both definitely fall into online social network category. This suggested that this question is non-trivial and might be a good question to post to the free knowledge market sites for the purpose of a small empirical test. I was interested if participants of these sites will be able to suggest a site that is earlier than Classmates.com or explain why Classmates.com shouldn’t be counted as a social network site. This question also asks for problem reformulation “What are the freatures of onlien social network, in the first place?” and requires some research. At the same time it is possible to verify the answers by tracing their references. So here is the question I posted into 3form, Naver, Yahoo, Mail.ru, and Google services:
What was the first online social network?
I am interested to know what was the name of the first online social network, who implemented it, and when. Thank you for your answers!
3form Free Knowledge Exchange:
- Wikipedia suggests that it was Classmates.com (1995). Maybe Email (with address book) can be thought of the frist online social network. Email existed since 1972, though I am not sure when the first email address books were implemented.
- A social network is a social structure made of connected individuals. This definition suggests that the Internet itself is the first online social network.
- “social networking on the PC based internet preceed the internet on PCs itself (having existed on “walled garden” BBS systems of that time such as Compuserve, AOL, Genie and Prodigy before they connected to the mostly then university and government used internet) being roughly 16-17 years of age as a mass market proposition (obviously early social networking existed on mainframes” link
- I would say it was The Well. http://en.wikipedia.org/wiki/WELL
Naver Knowledge-iN:
No answers were received from Naver.
Yahoo Answers:
- I believe it was AOL. Ok, probably not, but that was the first commercially availible one.
- Of course it depends on how you define the term “social networking site”.
- The worldwide distributed discussion system which is known as usenet (but whose proper name is netnews) was developed in 1979 by Steve Bellovin, Tom Truscott, and Jim Ellis. I personally believe this netnews as the first social networking site — it was certainly the first that relied on the
Internet - although it also used a transport called UUCP for unix-to-unix-copy.
Mail.ru Answers (translated from Russian):
- IMHO it is LiveJournal
Google.ru Answers (translated from Russian):
- Cites a russian article “The first social net of the Internet”: “Who was the first? Different sources mention different social networks, however the historical records give unambiguous answer—the first social network appears in the internet in 1995. The website of social network Classmates.com was opened for users in 1995 by Randy Conrad, the owner of Classmates Online, Inc. The website helped registered users to find and keep connections with their friends and contacts, with whom one had relationships throughout their life—preschool, school, colledge, work, and military service. Now Classmates.com has more than 40 millions registered users from the US and Canada.”
- In addition to the previous answer. … The term “social network” was introduced by sociologist James Barnes from Manchester School in his work “Classes and …” (This long answer lists the major social networks in the US and Russia and citing a jornal article “Social Networks in the Internet” from ComputerPress).
- (Third answer lists networks corrently popular in Italy, Latvia, Estonia and seems irrelevant to the question).
Yahoo was the fastest to provide answers. Google was also very fast and also provided the largest number of answers in the first day (3). Two answers from Yahoo and two answers from Google were received in less than 15 minutes after the question was posted. The remaining answers came within one day, with no answers for subsequent days, despite the fact that any question is open for answering for 4 days in Yahoo Answers, 5 days in Google answers, and no time limit in 3form (the question is kept if at least one of the participants is interested in keeping it). I can speculate that Yahoo and Google use recency of the question as a criterion to determine its salience/exposure, this would allow to direct the most of the answerers’ attention towards recent questions and receive answers to new and easy questions faster at the expense of older and more difficult questons. If the problem is not solved within one day, my experience is that it is unlikely to be solved in successive days in Yahoo/Google services unless it is reposted again. In this situation, multiple reposts will be needed to answer a difficult question and on each repost the problem solving process will start from scratch without benefitting from older solutions (you might post a link to the old thread in the question to compensate this). These services seem to be a good way to answer simple questions quickly, but it seems not suitable for solving problems that are somewhat more difficult. 3form, on the opposite, seems more suitable to address more difficult problems that are unlikely be answered in one day. Of course more experimentation and research is necessary to arrive at some reliable generalizations. This post is just a first step in this direction.
Conclusions
As suggested by their name “Answers” services are more appropriate for questions that are easy to answer and especially when the answer is instantly needed. They should be your second choice after doing search on wikipedia or on the web. If the question is not answered within one day, it is unlikely to be answered. A good idea is to post it again (maybe at a different site). Most of the answers at these sites arrive within minutes after the posting. If your problem requires some time, research, and/or creativity, you might have better chances at 3form that gives people more time to find solutions. 3form is also preferrable when (1) you have ill-defined problem that often needs reformulation and/or making assumptions, (2) many other people might be interested in the same problem, or (3) you lack expertise to select the best solution out of many (other services have strong biases in solution evaluation that often prevent them from selecting the best solutions). Korean knowledge markets represented here by Naver Knowledge iN offer the richest set of features. Russian Mail.ru has the most intricate incentive system. In our small test they were not particularly helpful, but their distinctive features seems quite useful.
In summary, if you need to find certain knowledge in English I would recommend the following sequence of steps: (1) Wikipedia search, (2) Google web search, (3) Yahoo Answers, (4) 3form. Each following step requires significantly more time than the previous step. This might change if the Google will make its new Q&A service available in English.
Acknowledgments: This text benefitted greatly from the help of Hwanjo Yu and Sang-Chul Lee in collecting information on Korean knowledge markets that is not available in English.