Kerckhoffs's principle (also called Kerckhoffs's desideratum, assumption, axiom, doctrine or law) of cryptography was stated by the Dutch cryptographer Auguste Kerckhoffs in the 19th century. The principle holds that a cryptosystem should be secure, even if everything about the system, except the key, is public knowledge. This concept is widely embraced by cryptographers, in contrast to security through obscurity, which is not. Kerckhoffs's principle was phrased by the American mathematician Claude Shannon as "the enemy knows the system", i.e., "one ought to design systems under the assumption that the enemy will immediately gain full familiarity with them". In that form, it is called Shannon's maxim. Another formulation by American researcher and professor Steven M. Bellovin is: In other words—design your system assuming that your opponents know it in detail. (A former official at NSA's National Computer Security Center told me that the standard assumption there was that serial number 1 of any new device was delivered to the Kremlin.) == Origins == The invention of telegraphy radically changed military communications and increased the number of messages that needed to be protected from the enemy dramatically, leading to the development of field ciphers which had to be easy to use without large confidential codebooks prone to capture on the battlefield. It was this environment which led to the development of Kerckhoffs's requirements. Auguste Kerckhoffs was a professor of German language at Ecole des Hautes Etudes Commerciales (HEC) in Paris. In early 1883, Kerckhoffs's article, La Cryptographie Militaire, was published in two parts in the Journal of Military Science, in which he stated six design rules for military ciphers. Translated from French, they are: The system must be practically, if not mathematically, indecipherable; It should not require secrecy, and it should not be a problem if it falls into enemy hands; It must be possible to communicate and remember the key without using written notes, and correspondents must be able to change or modify it at will; It must be applicable to telegraph communications; It must be portable, and should not require several persons to handle or operate; Lastly, given the circumstances in which it is to be used, the system must be easy to use and should not be stressful to use or require its users to know and comply with a long list of rules. Some are no longer relevant given the ability of computers to perform complex encryption. The second rule, now known as Kerckhoffs's principle, is still critically important. == Explanation of the principle == Kerckhoffs viewed cryptography as a rival to, and a better alternative than, steganographic encoding, which was common in the nineteenth century for hiding the meaning of military messages. One problem with encoding schemes is that they rely on humanly-held secrets such as "dictionaries" which disclose for example, the secret meaning of words. Steganographic-like dictionaries, once revealed, permanently compromise a corresponding encoding system. Another problem is that the risk of exposure increases as the number of users holding the secrets increases. Nineteenth century cryptography, in contrast, used simple tables which provided for the transposition of alphanumeric characters, generally given row-column intersections which could be modified by keys which were generally short, numeric, and could be committed to human memory. The system was considered "indecipherable" because tables and keys do not convey meaning by themselves. Secret messages can be compromised only if a matching set of table, key, and message falls into enemy hands in a relevant time frame. Kerckhoffs viewed tactical messages as only having a few hours of relevance. Systems are not necessarily compromised, because their components (i.e. alphanumeric character tables and keys) can be easily changed. === Advantage of secret keys === Using secure cryptography is supposed to replace the difficult problem of keeping messages secure with a much more manageable one, keeping relatively small keys secure. A system that requires long-term secrecy for something as large and complex as the whole design of a cryptographic system obviously cannot achieve that goal. It only replaces one hard problem with another. However, if a system is secure even when the enemy knows everything except the key, then all that is needed is to manage keeping the keys secret. There are a large number of ways the internal details of a widely used system could be discovered. The most obvious is that someone could bribe, blackmail, or otherwise threaten staff or customers into explaining the system. In war, for example, one side will probably capture some equipment and people from the other side. Each side will also use spies to gather information. If a method involves software, someone could do memory dumps or run the software under the control of a debugger in order to understand the method. If hardware is being used, someone could buy or steal some of the hardware and build whatever programs or gadgets needed to test it. Hardware can also be dismantled so that the chip details can be examined under the microscope. === Maintaining security === A generalization some make from Kerckhoffs's principle is: "The fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security." Bruce Schneier ties it in with a belief that all security systems must be designed to fail as gracefully as possible: Kerckhoffs's principle applies beyond codes and ciphers to security systems in general: every secret creates a potential failure point. Secrecy, in other words, is a prime cause of brittleness—and therefore something likely to make a system prone to catastrophic collapse. Conversely, openness provides ductility. Any security system depends crucially on keeping some things secret. However, Kerckhoffs's principle points out that the things kept secret ought to be those least costly to change if inadvertently disclosed. For example, a cryptographic algorithm may be implemented by hardware and software that is widely distributed among users. If security depends on keeping that secret, then disclosure leads to major logistic difficulties in developing, testing, and distributing implementations of a new algorithm – it is "brittle". On the other hand, if keeping the algorithm secret is not important, but only the keys used with the algorithm must be secret, then disclosure of the keys simply requires the simpler, less costly process of generating and distributing new keys. == Applications == In accordance with Kerckhoffs's principle, the majority of civilian cryptography makes use of publicly known algorithms. By contrast, ciphers used to protect classified government or military information are often kept secret (see Type 1 encryption). However, it should not be assumed that government/military ciphers must be kept secret to maintain security. It is possible that they are intended to be as cryptographically sound as public algorithms, and the decision to keep them secret is in keeping with a layered security posture. == Security through obscurity == It is moderately common for companies to keep the inner workings of a system secret. Some argue this "security by obscurity" makes the product safer and less vulnerable to attack. A counter-argument is that keeping the innards secret may improve security in the short term, but in the long run, only systems that have been published and analyzed should be trusted. Steven Bellovin and Randy Bush commented: Security Through Obscurity Considered Dangerous Hiding security vulnerabilities in algorithms, software, and/or hardware decreases the likelihood they will be repaired and increases the likelihood that they can and will be exploited. Discouraging or outlawing discussion of weaknesses and vulnerabilities is extremely dangerous and deleterious to the security of computer systems, the network, and its citizens. Open Discussion Encourages Better Security The long history of cryptography and cryptoanalysis has shown time and time again that open discussion and analysis of algorithms exposes weaknesses not thought of by the original authors, and thereby leads to better and more secure algorithms. As Kerckhoffs noted about cipher systems in 1883 [Kerc83], "Il faut qu'il n'exige pas le secret, et qu'il puisse sans inconvénient tomber entre les mains de l'ennemi." (Roughly, "the system must not require secrecy and must be able to be stolen by the enemy without causing trouble.")
Logogen model
The logogen model of 1969 is a model of speech recognition that uses units called "logogens" to explain how humans comprehend spoken or written words. Logogens are a vast number of specialized recognition units, each able to recognize one specific word. This model provides for the effects of context on word recognition. == Overview == The word logogen can be traced back to the Greek-language word logos, which means "word", and genus, which means "birth". British scientist John Morton's logogen model was designed to explain word recognition using a new type of unit known as a logogen. A critical element of this theory is the involvement of lexicons, or specialized aspects of memory that include semantic and phonemic information about each item that is contained in memory. A given lexicon consists of many smaller, abstract items known as logogens. Logogens contain a variety of properties about given word such as their appearance, sound, and meaning. Logogens do not store words within themselves, but rather they store information that is specifically necessary for retrieval of whatever word is being searched for. A given logogen will become activated by psychological stimuli or contextual information (words) that is consistent with the properties of that specific logogen and when the logogen's activation level rises to or above its threshold level, the pronunciation of the given word is sent to the output system. Certain stimuli can affect the activation levels of more than one word at a time, usually involving words that are similar to one another. When this occurs, whichever of the words' activation levels reaches the threshold level, it is that word that is then sent to the output system with the subject remaining unaware of any partially excited logogens. This assumption was made by Marslen-Wilson and Welch (1978), who added to the model some assumptions of their own in order to account for their experimental results. They also assumed that the analysis of phonetic input can only become available to other parts of the system by process of how the input affects the logogen system. Finally, Marslen-Wilson and Welch assume that the first syllable of a given word will increase the activation level of a given logogen more than those of the latter syllables, which supported the data found at the time. == Analysis == The logogen model can be used to help linguists explain particular occurrences in the human language. The most-helpful application of the model is to show how one accesses words and their meanings in the lexicon. The word-frequency effect is best explained by the logogen model in that words (or logogens) that have a higher frequency (or are more common) have a lower threshold. This means that they require less perceptual power in the brain to be recognized and decoded from the lexicon and are recognized faster than those words that are less common. Also, with high-frequency words, the recovery from lowering the item's threshold is less fulfilled compared to low-frequency words so less sensory information is needed for that particular item's recognition. There are ways to lower thresholds, such as repetition and semantic priming. Also, each time a word is encountered through these methods, the threshold for that word is temporarily lowered partially because of its recovering ability. This model also conveys that specific concrete words are recalled better because they use images and logogens, whereas abstract words are not as easily recalled well because they only use logogens, hence showing the difference in thresholds between these two types of words. At the time of its conception, Morton's logogen model was one of the most influential models in springing up other parallel word access models and served as the essential basis for these subsequent models. Morton's model also strongly influenced other contemporary theories on lexical access. However, despite the advantages that the logogen theory presents, it also displays some negative facets. First and foremost, the logogen model does not explain all occurrences in language, such as the introduction of new words or non-words into a person's lexicon. Also, because of the distinctive model application, it may vary in its effectiveness in different languages. == Criticisms == While this model does a reasonable job of understanding the underlying semantics of many aspects in psycholinguistics, there are some flaws that have been pointed out in the logogen model. It has been argued that the prior stimulus patterns that have been seen in the logogen theory are not centrally localized in the logogen itself but are actually distributed throughout the different pathways over which the stimulus is being processed. What this directs at is that the notion and proliferation of logogens was due to modality. In essence, the logogen is unnecessary in the idea of attaining the title of being a recognition unit because of the variety of pathways that it is open to, not just logogens. Another criticism has been that this model essentially ignores larger and more critical structures in language and phonetics such as the different syntactic rules or grammatical construction that innately exists in language. Since this model overtly limits itself to the scope of lexical access then this model is seen as biased and misunderstood. To many psychologists, the logogen model does not meet the functional or representational adequacy that a theory should include to sufficiently comprehend language. Also, another criticism is that the logogen theory was supposed to predict that stimulus degradation should affect priming and word frequency in humans. However, many psychologists have conducted studies and researched the model to show that only priming and not word frequency is interacted with stimulus degradation. Priming is supposed to deteriorate a stimulus because it postulates that the semantic characteristics of previously known words are fed back into the detector of a person which in turn raises the threshold of related items. In word frequency, stimulus degradation is supposed to occur because it postulates that familiar words have lower thresholds than their low-frequency counterparts. However, in studies, priming is the only structure that does show observable and notable stimulus decadence. Even though the logogen theory has many unfilled holes, Morton was a revolutionary of his field whose speculation and research has opened up a remarkable era of psycholinguistics. == Other models to consider == cohort model – This model was proposed by Marslen-Wilson and was designed specifically to account for auditory word recognition. It works by breaking the word down and states that when a word is heard all words that begin with the first sound of the target word are activated. This set of words is considered the cohort. Once the first cohort has been activated, the other information, or sounds in the word narrow down the choices. The person recognizes the word when you are left with a single choice; this is considered the "recognition point". checking model – This model was developed by Norris in 1986. In this particular model, he took the approach that any word that partially matches the input is analyzed and checked to see if it fits with the context of the situation. interactive-activation model – This model is considered a connectionist model. Proposed by McClelland and Rumelhart in the 1981 to 1982 period, it is based around nodes, which are visual features, and positions of letters within a given word. They also act as word detectors which have inhibitory and excitatory connections between them. This model starts with first letter and suggests that all the words with that first letter are activated at first and then going through the word one can determine what the word is they are looking at. The main principle is that mental phenomena can be described by interconnected networks of simple units. verification model – The model was developed by Curtis Becker in 1970. The main idea is that a small number of candidates that are activated in parallel are subject to a serial-verification process. This model starts the word-recognition process with a basic representation of the stimulus. Then, sensory trace, consisting of line features is used to activate word detectors. When an acceptable number of detectors are activated these are used to generate a search set. These items are drawn from the lexicon on the basis of similarity to the sensory trace, which help with the identity of the stimulus. Then, in a serial process the candidates are compared to the representation of the sensory-trace input. == Related concepts == word frequency – This is the belief that the speed and accuracy with which a word is recognized is related to how frequently the word occurs in our language. Each logogen has a threshold (for identification) and words with higher frequencies have lower thresholds. Words with higher freq
Representational harm
Systems cause representational harm when they misrepresent a group of people in a negative manner. Representational harms include perpetuating harmful stereotypes about or minimizing the existence of a social group, such as a racial, ethnic, gender, or religious group. Machine learning algorithms often commit representational harm when they learn patterns from data that have algorithmic bias, and this has been shown to be the case with large language models. While preventing representational harm in models is essential to prevent harmful biases, researchers often lack precise definitions of representational harm and conflate it with allocative harm, an unequal distribution of resources among social groups, which is more widely studied and easier to measure. However, recognition of representational harms is growing and preventing them has become an active research area. Researchers have recently developed methods to effectively quantify representational harm in algorithms, making progress on preventing this harm in the future. == Types == Three prominent types of representational harm include stereotyping, denigration, and misrecognition. These subcategories present many dangers to individuals and groups. Stereotypes are oversimplified and usually undesirable representations of a specific group of people, usually by race and gender. This often leads to the denial of educational, employment, housing, and other opportunities. For example, the model minority stereotype of Asian Americans as highly intelligent and good at mathematics can be damaging professionally and academically. Representational harm happens when the representation of details teams improves damaging stereotypes, developing social exclusion and prejudice. This experience is particularly noticeable in the depiction of marginalised groups, containing people of color, women, LGBTQ+ people, and people with handicaps. Media depictions of these groups generally stop working to catch their array and intricacy. Instead, they are typically reduced to one-dimensional caricatures, which ultimately continue social prejudices. These organised depictions contribute to the help of hazardous stereotypes and the marginalisation of these locations. Denigration is the action of unfairly criticizing individuals. This frequently happens when the demeaning of social groups occurs. For example, when searching for "Black-sounding" names versus "white-sounding" ones, some retrieval systems bolster the false perception of criminality by displaying ads for bail-bonding businesses. A system may shift the representation of a group to be of lower social status, often resulting in a disregard from society. Research shows that hazardous depictions in the media can have substantial emotional and social impacts on both individuals and areas. Lawrence Bobo examined the issue of Ethnic stereotype in film, tv, and marketing. African Americans are commonly received duties specified by features such as "violent tendencies," "laziness," or being "merely for contentment features." While these representations might appear varied externally, they stay to boost underlying frameworks of white prominence and racial inequality. As a circumstances, Black individuals are frequently represented as law offenders or in secondary roles, which adds to the support of Ethnic stereotype and Institutional racism. Misrecognition, or incorrect recognition, can display in many forms, including, but not limited to, erasing and alienating social groups, and denying people the right to self-identify. Erasing and alienating social groups involves the unequal visibility of certain social groups; specifically, systematic ineligibility in algorithmic systems perpetuates inequality by contributing to the underrepresentation of social groups. Not allowing people to self-identify is closely related as people's identities can be 'erased' or 'alienated' in these algorithms. Misrecognition causes more than surface-level harm to individuals: psychological harm, social isolation, and emotional insecurity can emerge from this subcategory of representational harm. == Quantification == As the dangers of representational harm have become better understood, some researchers have developed methods to measure representational harm in algorithms. Modeling stereotyping is one way to identify representational harm. Representational stereotyping can be quantified by comparing the predicted outcomes for one social group with the ground-truth outcomes for that group observed in real data. For example, if individuals from group A achieve an outcome with a probability of 60%, stereotyping would be observed if it predicted individuals to achieve that outcome with a probability greater than 60%. The group modeled stereotyping in the context of classification, regression, and clustering problems, and developed a set of rules to quantitatively determine if the model predictions exhibit stereotyping in each of these cases. Other attempts to measure representational harms have focused on applications of algorithms in specific domains such as image captioning, the act of an algorithm generating a short description of an image. In a study on image captioning, researchers measured five types of representational harm. To quantify stereotyping, they measured the number of incorrect words included in the model-generated image caption when compared to a gold-standard caption. They manually reviewed each of the incorrectly included words, determining whether the incorrect word reflected a stereotype associated with the image or whether it was an unrelated error, which allowed them to have a proxy measure of the amount of stereotyping occurring in this caption generation. These researchers also attempted to measure demeaning representational harm. To measure this, they analyzed the frequency with which humans in the image were mentioned in the generated caption. It was hypothesized that if the individuals were not mentioned in the caption, then this was a form of dehumanization. == Examples == One of the most notorious examples of representational harm was committed by Google in 2015 when an algorithm in Google Photos classified Black people as gorillas. Developers at Google said that the problem was caused because there were not enough faces of Black people in the training dataset for the algorithm to learn the difference between Black people and gorillas. Google issued an apology and fixed the issue by blocking its algorithms from classifying anything as a primate. In 2023, Google's photos algorithm was still blocked from identifying gorillas in photos. Another prevalent example of representational harm is the possibility of stereotypes being encoded in word embeddings, which are trained using a wide range of text. These word embeddings are the representation of a word as an array of numbers in vector space, which allows an individual to calculate the relationships and similarities between words. However, recent studies have shown that these word embeddings may commonly encode harmful stereotypes, such as the common example that the phrase "computer programmer" is oftentimes more closely related to "man" than it is to "women" in vector space. This could be interpreted as a misrepresentation of computer programming as a profession that is better performed by men, which would be an example of representational harm. == Addressing representational harm == Initiatives to minimise representational harm include advertising for even more inclusive and accurate portrayals of marginalised teams in the media. Scholars and protestors recommend that the method to reducing representational injury depends on raising the selection of voices both behind and before the digital video camera. When marginalized groups are provided the chance to represent themselves, they can check traditional stereotypes and present their experiences additional authentically. Over the last few years, efforts to increase representation of people of color, women, and LGBTQ+ people in conventional media have made some progression. Films such as Selma, routed by Ava DuVernay, and tv series like Pose, developed by Ryan Murphy, have actually been extensively applauded for their nuanced and respectful representations of marginalised communities. These tasks existing complex individualities and stories that move past streamlined stereotypes. Self-representation is one more crucial method to addressing representational harm. By equipping marginalised locations to create their really own tales, media designers can effectively reduce the perpetuation of hazardous stereotypes. This procedure consists of both the manufacturing of media product by participants of these communities and proactively difficult typical media structures that have actually historically omitted them.
Predictions of the end of Wikipedia
Various observers have predicted the end of Wikipedia since it rose to prominence, with potential pitfalls from lack of quality-control, artificial intelligence or inconsistencies among contributors. Alternative online encyclopedias have been proposed as replacements for Wikipedia, including WolframAlpha, as well as the both now-defunct Knol (from Google) and Owl (from AOL). A 2013 review raised alarms regarding Wikipedia's shortcomings on hoaxes, on vandalism, an imbalance of material, and inadequate quality control of articles. Earlier critiques lamented the vulgar content and absence of sufficient references in articles. Others suggest that the unwarranted deletion of useful articles from Wikipedia may portend its end, which itself inspired the creation of the now inactive Deletionpedia. Contrary to such predictions, Wikipedia has constantly grown in both size and influence. Recent developments with artificial intelligence in Wikimedia projects have prompted new predictions that AI applications, which consume free and open content, will replace Wikipedia. == Personnel == Wikipedia is crowdsourced by a few million volunteer editors. Of the millions of registered editors, only tens of thousands contribute the majority of its contents, and a few thousand do quality control and maintenance work. As the encyclopedia expanded in the 2010s, the number of active editors did not grow proportionately. Various sources predicted that Wikipedia will eventually have too few editors to be functional and collapse from lack of participation. English Wikipedia has 818 volunteer administrators who perform various functions, including functions similar to those carried out by a forum moderator. Critics have described their actions as harsh, bureaucratic, biased, unfair, or capricious and predicted that the resulting outrage would lead to the site's closure. Various 2012 articles reported that a decline in English Wikipedia's recruitment of new administrators could end Wikipedia. === Decline in editors (2014–2015) === A 2014 trend analysis published in The Economist stated that "The number of editors for the English-language version has fallen by a third in seven years." The attrition rate for active editors in English Wikipedia was described by The Economist as substantially higher than in other (non-English) Wikipedias. It reported that in other languages, the number of "active editors" (those with at least five edits per month) has been relatively constant since 2008: some 42,000 editors, with narrow seasonal variances of about 2,000 editors up or down. In the English Wikipedia, the number of active editors peaked in 2007 at about 50,000 editors, and fell to 30,000 editors in 2014. Given that the trend analysis published in The Economist presented the number of active editors for non-English Wikipedias as remaining relatively constant, sustaining their numbers at approximately 42,000 active editors, the contrast pointed to the effectiveness of Wikipedia in those languages to retain their active editors on a renewable and sustained basis. Though different language versions of Wikipedia have different policies, no comment identified a particular policy difference as potentially making a difference in the rate of editor attrition for English Wikipedia. Editor count showed a slight uptick a year later, and no clear trend after that. In a 2013 article, Tom Simonite of MIT Technology Review said that for several years running, the number of Wikipedia editors had been falling, and cited the bureaucratic structure and rules as a factor. Simonite alleged that some Wikipedians use the labyrinthine rules and guidelines to dominate others and have a vested interest in keeping the status quo. A January 2016 article in Time by Chris Wilson said Wikipedia might lose many editors because a collaboration of occasional editors and smart software will take the lead. Andrew Lih and Andrew Brown both maintain editing Wikipedia with smartphones is difficult and discourages new potential contributors. Lih alleges there is serious disagreement among existing contributors on how to resolve this. In 2015, Lih feared for Wikipedia's long-term future while Brown feared problems with Wikipedia would remain and rival encyclopedias would not replace it. == Viewers and fundraisers == As of 2015, with more viewing by smartphones, there had been a marked decline in persons who viewed Wikipedia from their computers, and according to The Washington Post "[people are] far less likely to donate". At the time, the Wikimedia Foundation reported reserves equivalent to one year's budgeted expenditures. On the other hand, the number of paid staff had ballooned, so those expenses increased. In 2021, Andreas Kolbe, a former co-editor-in-chief of The Signpost, wrote that the Wikimedia Foundation was reaching its 10-year goal of a US$100 million endowment, five years earlier than planned, which may surprise donors and users around the world who regularly see Wikipedia fundraising banners. He also said accounting methods disguise the size of operating surpluses, top managers earn $300,000 – 400,000 a year, and over 40 people work exclusively on fundraising. == Artificial intelligence == Wikipedia faces a decline in human visitors, raising concerns about its long-term sustainability and community participation. The Wikimedia Foundation (WMF), when reporting this decline, attributed this in part to the lack of clicks from users of large language models and search engines that are using content from Wikipedia. Data published in August 2025 showed that after the launch of ChatGPT and the rise of other AI-powered search summaries, some types of articles on Wikipedia — especially those that closely resemble the kind of content ChatGPT produces — experienced a noticeable drop in readership. Overall human pageviews reportedly fell by about 8% between 2024 and 2025, suggesting that AI-overviews and chatbots are increasingly being used in place of direct visits to Wikipedia. According to industry web analytics data, ChatGPT's estimated monthly web traffic surpassed that of Wikipedia since May 2025, as visits to ChatGPT continued to grow while Wikipedia’s total site traffic declined. == Timeline of predictions == On the eve of the 20th anniversary of Wikipedia, associate professor of the Department of Communication Studies at Northeastern University Joseph Reagle conducted a retrospective study of numerous "predictions of the ends of Wikipedia" over two decades, divided into chronological waves: "Early growth (2001–2002)", "Nascent identity (2001–2005)", "Production model (2005–2010)", "Contributor attrition (2009–2017)" and the current period "(2020–)". Each wave brought its distinctive fatal predictions, which never came true; as a result, Reagle concluded Wikipedia was not in danger. Concern grew in 2023 that the ubiquity and proliferation of artificial intelligence (AI) may adversely affect Wikipedia. Rapid improvements and widespread application of AI may render Wikipedia obsolete or reduce its importance. A 2023 study found that AI, when applied to Wikipedia, works most efficiently for error-correction, while Wikipedia still needs to be written by humans.
Resource Description Framework
The Resource Description Framework (RDF) is a method to describe and exchange graph data. It was originally designed as a data model for metadata by the World Wide Web Consortium (W3C). It provides a variety of syntax notations and formats, of which the most widely used is Turtle (Terse RDF Triple Language). RDF is a directed graph composed of triple statements. An RDF graph statement is represented by: (1) a node for the subject, (2) an arc from subject to object, representing a predicate, and (3) a node for the object. Each of these parts can be identified by a Internationalized Resource Identifier (IRI). An object can also be a literal value. This simple, flexible data model has a lot of expressive power to represent complex situations, relationships, and other things of interest, while also being appropriately abstract. RDF was adopted as a W3C recommendation in 1999. The RDF 1.0 specification was published in 2004, and the RDF 1.1 specification in 2014. SPARQL is a standard query language for RDF graphs. RDF Schema (RDFS), Web Ontology Language (OWL) and SHACL (Shapes Constraint Language) are ontology languages that are used to describe RDF data. == Overview == The RDF data model is similar to classical conceptual modeling approaches (such as entity–relationship or class diagrams). It is based on the idea of making statements about resources (in particular web resources) in expressions of the form subject–predicate–object, known as triples. The subject denotes the resource; the predicate denotes traits or aspects of the resource, and expresses a relationship between the subject and the object. For example, one way to represent the notion "The sky has the color blue" in RDF is as the triple: a subject denoting "the sky", a predicate denoting "has the color", and an object denoting "blue". Therefore, RDF uses subject instead of object (or entity) in contrast to the typical approach of an entity–attribute–value model in object-oriented design: entity (sky), attribute (color), and value (blue). RDF is an abstract model with several serialization formats (being essentially specialized file formats). In addition the particular encoding for resources or triples can vary from format to format. This mechanism for describing resources is a major component in the W3C's Semantic Web activity: an evolutionary stage of the World Wide Web in which automated software can store, exchange, and use machine-readable information distributed throughout the Web, in turn enabling users to deal with the information with greater efficiency and certainty. RDF's simple data model and ability to model disparate, abstract concepts has also led to its increasing use in knowledge management applications unrelated to Semantic Web activity. A collection of RDF statements intrinsically represents a labeled, directed multigraph. This makes an RDF data model better suited to certain kinds of knowledge representation than other relational or ontological models. As RDFS, OWL and SHACL demonstrate, one can build additional ontology languages upon RDF. == History == The initial RDF design, intended to "build a vendor-neutral and operating system- independent system of metadata", derived from the W3C's Platform for Internet Content Selection (PICS), an early web content labelling system, but the project was also shaped by ideas from Dublin Core, and from the Meta Content Framework (MCF), which had been developed during 1995 to 1997 by Ramanathan V. Guha at Apple and Tim Bray at Netscape. A first public draft of RDF appeared in October 1997, issued by a W3C working group that included representatives from IBM, Microsoft, Netscape, Nokia, Reuters, SoftQuad, and the University of Michigan. In 1999, the W3C published the first recommended RDF specification, the Model and Syntax Specification ("RDF M&S"). This described RDF's data model and an XML serialization. Two persistent misunderstandings about RDF developed at this time: firstly, due to the MCF influence and the RDF "Resource Description" initialism, the idea that RDF was specifically for use in representing metadata; secondly that RDF was an XML format rather than a data model, and only the RDF/XML serialisation being XML-based. RDF saw little take-up in this period, but there was significant work done in Bristol, around ILRT at Bristol University and HP Labs, and in Boston at MIT. RSS 1.0 and FOAF became exemplar applications for RDF in this period. The recommendation of 1999 was replaced in 2004 by a set of six specifications: "The RDF Primer", "RDF Concepts and Abstract", "RDF/XML Syntax Specification (revised)", "RDF Semantics", "RDF Vocabulary Description Language 1.0", and "The RDF Test Cases". This series was superseded in 2014 by the following six "RDF 1.1" documents: "RDF 1.1 Primer", "RDF 1.1 Concepts and Abstract Syntax", "RDF 1.1 XML Syntax", "RDF 1.1 Semantics", "RDF Schema 1.1", and "RDF 1.1 Test Cases". == RDF topics == === Vocabulary === The vocabulary defined by the RDF specification is as follows: ==== Classes ==== ===== rdf ===== rdf:XMLLiteral the class of XML literal values rdf:Property the class of properties rdf:Statement the class of RDF statements rdf:Alt, rdf:Bag, rdf:Seq containers of alternatives, unordered containers, and ordered containers (rdfs:Container is a super-class of the three) rdf:List the class of RDF Lists rdf:nil an instance of rdf:List representing the empty list ===== rdfs ===== rdfs:Resource the class resource, everything rdfs:Literal the class of literal values, e.g. strings and integers rdfs:Class the class of classes rdfs:Datatype the class of RDF datatypes rdfs:Container the class of RDF containers rdfs:ContainerMembershipProperty the class of container membership properties, rdf:_1, rdf:_2, ..., all of which are sub-properties of rdfs:member ==== Properties ==== ===== rdf ===== rdf:type an instance of rdf:Property used to state that a resource is an instance of a class rdf:first the first item in the subject RDF list rdf:rest the rest of the subject RDF list after rdf:first rdf:value idiomatic property used for structured values rdf:subject the subject of the RDF statement rdf:predicate the predicate of the RDF statement rdf:object the object of the RDF statement rdf:Statement, rdf:subject, rdf:predicate, rdf:object are used for reification (see below). ===== rdfs ===== rdfs:subClassOf the subject is a subclass of a class rdfs:subPropertyOf the subject is a subproperty of a property rdfs:domain a domain of the subject property rdfs:range a range of the subject property rdfs:label a human-readable name for the subject rdfs:comment a description of the subject resource rdfs:member a member of the subject resource rdfs:seeAlso further information about the subject resource rdfs:isDefinedBy the definition of the subject resource This vocabulary is used as a foundation for RDF Schema, where it is extended. === Serialization formats === Several common serialization formats are in use, including: Turtle, a compact, human-friendly format. TriG, an extension of Turtle to datasets. N-Triples, a very simple, easy-to-parse, line-based format that is not as compact as Turtle. N-Quads, a superset of N-Triples, for serializing multiple RDF graphs. JSON-LD, a JSON-based serialization. N3 or Notation3, a non-standard serialization that is very similar to Turtle, but has some additional features, such as the ability to define inference rules. RDF/XML, an XML-based syntax that was the first standard format for serializing RDF. RDF/JSON, an alternative syntax for expressing RDF triples using a simple JSON notation. RDF/XML is sometimes misleadingly called simply RDF because it was introduced among the other W3C specifications defining RDF and it was historically the first W3C standard RDF serialization format. However, it is important to distinguish the RDF/XML format from the abstract RDF model itself. Although the RDF/XML format is still in use, other RDF serializations are now preferred by many RDF users, both because they are more human-friendly, and because some RDF graphs are not representable in RDF/XML due to restrictions on the syntax of XML QNames. With a little effort, virtually any arbitrary XML may also be interpreted as RDF using GRDDL (pronounced 'griddle'), Gleaning Resource Descriptions from Dialects of Languages. RDF triples may be stored in a type of database called a triplestore. === Resource identification === The subject of an RDF statement is either a uniform resource identifier (URI) or a blank node, both of which denote resources. Resources indicated by blank nodes are called anonymous resources. They are not directly identifiable from the RDF statement. The predicate is a URI which also indicates a resource, representing a relationship. The object is a URI, blank node or a Unicode string literal. As of RDF 1.1 resources are identified by Internationalized Resource Identifiers (IRIs); IRIs are a generalization of URIs. In Semantic Web applications, and in re
Apache Pig
Apache Pig is a high-level platform for creating programs that run on Apache Hadoop. The language for this platform is called Pig Latin. Pig can execute its Hadoop jobs in MapReduce, Apache Tez, or Apache Spark. Pig Latin abstracts the programming from the Java MapReduce idiom into a notation which makes MapReduce programming high level, similar to that of SQL for relational database management systems. Pig Latin can be extended using user-defined functions (UDFs) which the user can write in Java, Python, JavaScript, Ruby or Groovy and then call directly from the language. == History == Apache Pig was originally developed at Yahoo Research around 2006 for researchers to have an ad hoc way of creating and executing MapReduce jobs on very large data sets. In 2007, it was moved into the Apache Software Foundation. === Naming === Regarding the naming of the Pig programming language, the name was chosen arbitrarily and stuck because it was memorable, easy to spell, and for novelty. The story goes that the researchers working on the project initially referred to it simply as 'the language'. Eventually they needed to call it something. Off the top of his head, one researcher suggested Pig, and the name stuck. It is quirky yet memorable and easy to spell. While some have hinted that the name sounds coy or silly, it has provided us with an entertaining nomenclature, such as Pig Latin for the language, Grunt for the shell, and PiggyBank for the CPAN-like shared repository. == Example == Below is an example of a "Word Count" program in Pig Latin: The above program will generate parallel executable tasks which can be distributed across multiple machines in a Hadoop cluster to count the number of words in a dataset such as all the webpages on the internet. == Pig vs SQL == In comparison to SQL, Pig has a nested relational model, uses lazy evaluation, uses extract, transform, load (ETL), is able to store data at any point during a pipeline, declares execution plans, supports pipeline splits, thus allowing workflows to proceed along DAGs instead of strictly sequential pipelines. On the other hand, it has been argued DBMSs are substantially faster than the MapReduce system once the data is loaded, but that loading the data takes considerably longer in the database systems. It has also been argued RDBMSs offer out of the box support for column-storage, working with compressed data, indexes for efficient random data access, and transaction-level fault tolerance. Pig Latin is procedural and fits very naturally in the pipeline paradigm while SQL is instead declarative. In SQL users can specify that data from two tables must be joined, but not what join implementation to use (You can specify the implementation of JOIN in SQL, thus "... for many SQL applications the query writer may not have enough knowledge of the data or enough expertise to specify an appropriate join algorithm."). Pig Latin allows users to specify an implementation or aspects of an implementation to be used in executing a script in several ways. In effect, Pig Latin programming is similar to specifying a query execution plan, making it easier for programmers to explicitly control the flow of their data processing task. SQL is oriented around queries that produce a single result. SQL handles trees naturally, but has no built in mechanism for splitting a data processing stream and applying different operators to each sub-stream. Pig Latin script describes a directed acyclic graph (DAG) rather than a pipeline. Pig Latin's ability to include user code at any point in the pipeline is useful for pipeline development. If SQL is used, data must first be imported into the database, and then the cleansing and transformation process can begin.
Business rule management system
A BRMS or business rule management system is a software system used to define, deploy, execute, monitor and maintain the variety and complexity of decision logic that is used by operational systems within an organization or enterprise. This logic, also referred to as business rules, includes policies, requirements, and conditional statements that are used to determine the tactical actions that take place in applications and systems. == Overview == A BRMS includes, at minimum: A repository, allowing decision logic to be externalized from core application code Tools, allowing both technical developers and business experts to define and manage decision logic A runtime environment, allowing applications to invoke decision logic managed within the BRMS and execute it using a business rules engine The top benefits of a BRMS include: Reduced or removed reliance on IT departments for changes in live systems. Although, QA and Rules testing would still be needed in any enterprise system. Increased control over implemented decision logic for compliance and better business management including audit logs, impact simulation and edit controls. The ability to express decision logic with increased precision, using a business vocabulary syntax and graphical rule representations (decision tables, decision models, trees, scorecards and flows) Improved efficiency of processes through increased decision automation. Some disadvantages of the BRMS include: Extensive subject matter expertise can be required for vendor specific products. In addition to appropriate design practices (such as Decision Modeling), technical developers must know how to write rules and integrate software with existing systems Poor rule harvesting approaches can lead to long development cycles, though this can be mitigated with modern approaches like the Decision Model and Notation (DMN) standard. Integration with existing systems is still required and a BRMS may add additional security constraints. Reduced IT department reliance may never be a reality due to continued introduction to new business rule considerations or object model perturbations The coupling of a BRMS vendor application to the business application may be too tight to replace with another BRMS vendor application. This can lead to cost to benefits issues. The emergence of the DMN standard has mitigated this to some degree. Most BRMS vendors have evolved from rule engine vendors to provide business-usable software development lifecycle solutions, based on declarative definitions of business rules executed in their own rule engine. BRMSs are increasingly evolving into broader digital decisioning platforms that also incorporate decision intelligence and machine learning capabilities. However, some vendors come from a different approach (for example, they map decision trees or graphs to executable code). Rules in the repository are generally mapped to decision services that are naturally fully compliant with the latest SOA, Web Services, or other software architecture trends. == Related software approaches == In a BRMS, a representation of business rules maps to a software system for execution. A BRMS therefore relates to model-driven engineering, such as the model-driven architecture (MDA) of the Object Management Group (OMG). It is no coincidence that many of the related standards come under the OMG banner. A BRMS is a critical component for Enterprise Decision Management as it allows for the transparent and agile management of the decision-making logic required in systems developed using this approach. == Associated standards == The OMG Decision Model and Notation standard is designed to standardize elements of business rules development, specially decision table representations. There is also a standard for a Java Runtime API for rule engines JSR-94. OMG Business Motivation Model (BMM): A model of how strategies, processes, rules, etc. fit together for business modeling OMG SBVR: Targets business constraints as opposed to automating business behavior OMG Production Rule Representation (PRR): Represents rules for production rule systems that make up most BRMS' execution targets OMG Decision Model and Notation (DMN): Represents models of decisions, which are typically managed by a BRMS RuleML provides a family of rule mark-up languages that could be used in a BRMS and with W3C RIF it provides a family of related rule languages for rule interchange in the W3C Semantic Web stack Many standards, such as domain-specific languages, define their own representation of rules, requiring translations to generic rule engines or their own custom engines. Other domains, such as PMML, also define rules.