File Syntax

Version 1.1 Working specification

• Introduction
• Definition of terms
• File syntax
• General features
• Character set
• White space
• End-of-line conventions
• Case sensitivity
• Implementation restrictions
• Maximum line length and character set
• Maximum data name, block code and frame code lengths
• Single-level loop constructions
• Non-expansion of save frame references
• Exclusion of global_ blocks
• Version identification
• Appendix A: A formal grammar for CIF
• Summary
• Explanation of the formal syntax
• Lexical tokens
• CIF grammar
• References

Introduction

1. This document describes the full syntax of the Crystallographic Information File (CIF).

Definition of terms

2. The following terms are used in the CIF specification documents with the specific meanings indicated here.

• 2.1. A CIF is a file conforming to the specification herein stated, containing either information on a crystallographic experiment or its results (or similar scientific content); or descriptions of the data identifiers in such a file.
• 2.2. A data file is understood to convey information relating to a crystallographic experiment.
• 2.3. A dictionary file is understood to contain information about the data items in one or more data files as identified by their data names.
• 2.4. A data name is a case-insensitive identifier (a string of characters beginning with an underscore character) of the content of an associated data value.
• 2.5. A data value is a string of characters representing a particular item of information. It may represent a single numerical value; a letter, word or phrase; extended discursive text; or in principle any coherent unit of data such as an image, audio clip or virtual-reality object.
• 2.6. A data item is a specific piece of information defined by a data name and an associated data value.
• 2.7. A tag is understood in this document to be a synonym for data name.
• 2.8. A data block is the highest-level component of a CIF, containing data items or save frames. A data block is identified by a data block header, which is an isolated character string (that is, bounded by white space and not forming part of a data value) beginning with the case-insensitive reserved characters data_.
• 2.9. A block code is the variable part of a data block header, e.g. the string foo in the header data_foo.
• 2.10. A save frame is a partitioned collection of data items within a data block, started by a save frame header, which is an isolated character string beginning with the case-insensitive reserved characters save_, and terminated with an isolated character string containing only the case-insensitive reserved characters save_.
• 2.11. A frame code is the variable part of a save frame header, e.g. the string foo in the header save_foo.

File syntax

3. The syntax of CIF is a proper subset of the syntax of STAR Files as described by Hall (1991) and Hall & Spadaccini (1994). The general structure is described below under the heading General features, and a number of subsections list specific restrictions to the STAR syntax that are in force within CIF. A formal language grammar using computer science notation is included as Appendix A.

General features

4. A CIF consists of data names (tags) and associated values organized into data blocks. A data block may contain data items (associated data names and data values) and/or it may contain save frames.

5. Save frames may only be used in dictionary files.

Implementation note: At a purely syntactic level there is no way to distinguish between dictionary and data files. (It is also to be noted that not all dictionary files contain save frames.) A fully validating parser must therefore be able to detect the start and termination of save frames, the uniqueness of the framecode within a data block, and the uniqueness of data names within a frame code. It is however legitimate for an application-based parser designed to handle only the contents of data files to consider the presence of a save frame as an error.

6. A data block begins with the reserved case-insensitive string data_ followed immediately by the name of the data block, forming a data block header. A save frame has a similar structure to a data block, but may not itself contain further save frames. A save frame begins with the reserved case-insensitive string save_ followed immediately by the name of the save frame, forming a save frame header. Unlike a data block, a save frame also has a marker for the end of the frame in the form of a repetition of the reserved case-insensitive word save_, this time without the name of the frame. Save frames may not nest. Within a single CIF, no two data blocks may have the same name; within a single data block no two save frames may have the same name, although a save frame may have the same name as a data block in the same CIF.

7. A given data name (tag) (see 2.4 and 2.7) may appear no more than once in a given data block or save frame. A tag may be followed by a single value, or a list of one or more tags may be marked by the preceding reserved case-insensitive word loop_ as the headings of the columns of a table of values. White space is used to separate a data block or save frame header from the contents of the data block or save frame, and to separate tags, values and the reserved word loop_. Data items (tags along with their associated values) that are not presented in a table of values may be relocated along with their values within the same data block or save frame without changing the meaning of the data block or save frame. Complete tables of values (the table column headings along with all columns of data) may be relocated within the same data block or save frame without changing the meaning of the data block or save frame. Within a table of values, each tag may be relocated along with its associated column of values within the same table of values without changing the meaning of the table of values. In general each row of a table of values may also be relocated within the same table of values without changing the meaning of the table of values. Combining tables of values or breaking up tables of values would change the meanings, and is likely to violate the rules for constructing such tables of values.

8. The case-insensitive word global_, used in STAR Files to introduce a group of data values with a scope extending to the end of the file, is an additional reserved word in CIF (that is, it may not be used as the unquoted value of any data item).

9. If a data value (see 2.5) contains white space or begins with a character string reserved for a special purpose, it must be delimited by one of several sets of special character strings (the choice of which is constrained if the data value contains characters interpretable as marking a new line of text according to the discussion in the following paragraphs). Such a data value will be indicated by the term non-simple data value.

10. A simple data value (i.e. one which does not contain white space or begin with a special character string) may optionally be delimited by any of the same set of delimiting character strings, except for data values that are to be interpreted as numbers.

11. The special character strings in this context are listed in the following table. The term "non-simple data values" in this table refers to data values beginning with these special character strings.

In addition the following case-insensitive reserved words may not occur as unquoted data values.

12. The complete syntactic description of a numeric data value is included in Appendix A (paragraph 57) under the production (i.e. rule for constructing a part of the language) <Numeric>.

13. The base CIF specification distinguishes between character and numeric values (see paragraph 15 of the document Common semantic features). Particular CIF applications may make more finely-grained distinctions within these types. The paragraphs immediately above have the corollary that a data value such as 12 that appears within a CIF may be quoted (e.g. '12') if, and only if it is to be interpreted and stored in computer memory as a character string and not a numeric value. For example '12' might legitimately appear as a label for an atomic site, where another alphabetic or alphanumeric string such as 'C12' is also acceptable; but it may not legitimately be used to represent an integer quantity twelve.

14. Matching single or double quote characters (' or ") may be used to bound a string representing a non-simple data value provided the string does not extend over more than one line.

15. Because data values are invariably separated from other tokens in the file by white space, such a quote-delimited character string may contain instances of the character used to delimit the string provided they are not followed by white space. For example, the data item

     _example  'a dog's life' 

16. Note that constructs such as 'an embedded \' quote' do not behave as in the case of many current programming languages; i.e. the backslash character in this context does not escape the special meaning of the delimiter character. A backslash preceding the apostrophe or double-quote characters does, however, have special meaning in the context of accented characters (paragraph 32 of the document Common semantic features) provided there is no white space immediately following the apostrophe or double-quote character.

17. The special sequence of end-of-line followed immediately by a semicolon in column one (denoted "<eol>;") may also be used as a delimiter at the beginning and end of a character string comprising a data value. The complete bounded string is called a text field, and may be used to convey multi-line values. The end-of-line associated with the closing semicolon does not form part of the data value. Within a multi-line text field, leading white space within text lines must be retained as part of the data value; trailing white space on a line may however be elided.

19. Matching square bracket characters, '[' and ']', are reserved for possible future introduction as delimiters of multi-line data values. At this revision of the CIF specification a data value may not begin with an unquoted left square bracket character '['. (While not strictly necessary, the right square bracket character ']' is restricted in the same way in recognition of its reserved use as a closing delimiter.)

;foo
;

; foo
  bar
;

21. A comment in a CIF begins with an unquoted character "#" and extends to the end of the current line.

Character set

22. Characters within a CIF are restricted to certain printable or white-space characters. Specifically, these are the ones located in the ASCII character set at decimal positions 09 (HT or horizontal tab), 10 (LF or line feed), 13 (CR or carriage return) and the letters, numerals and punctuation marks at positions 32-126.

White space

24. Any of the whitespace characters listed under the heading Character set (paragraph 22) (i.e. HT, LF, CR) and the visible space character SP (position number 32 in the ASCII encoding) may be used interchangeably to separate tokens, with the exception that the semicolon characters delimiting multiline text fields must be preceded by the whitespace character or characters understood as indicating an end of line (see next paragraph).

End-of-line conventions

25. The way in which a line is terminated is operating-system dependent. The STAR File specification does not address different operating-system conventions for encoding the end of a line of text in a text file. For a file generated and read in the same machine environment, this is rarely a problem, but increasingly applications on a network host may access files on different hosts through protocols designed to present a unified view of a file system. In practice, for current common operating systems many applications may regard the ASCII characters LF or CR or the sequence CR LF as signalling an end-of-line, inasmuch as these represent the end-of-line conventions supported under the common operating systems Unix, MacOS or DOS/Windows. On platforms with record-oriented operating systems applications must understand and implement the appropriate end-of-line convention. Care must be taken when transferring such files to other operating systems to insert the appropriate end-of-line characters for the target operating system. A more complete discussion is given in paragraph 42 below.

Case sensitivity

26. Data names, block and frames codes, and reserved words are case-insensitive. The case of any characters within data values must be respected.

Implementation restrictions

27. Certain allowed features of STAR File syntax have been expressly excluded or restricted from the CIF implementation.

Maximum line length and character set

28. Lines of text may not exceed 2048 characters in length. This count excludes the character or characters used by the operating system to mark the line termination.

The ASCII characters decimal 11 (VT) and 12 (FF) are excluded from the allowed character set (see paragraph 22).

Maximum data name, block code and frame code lengths

29. Data names may not exceed 75 characters in length.

30. Data block codes and save frame codes may not exceed 75 characters in length (and therefore data block headers and save frame headers may not exceed 80 characters in length).

Single-level loop constructs

31. Only a single level of looping is permitted.

Non-expansion of save frame references

32. Save frames are permitted in CIFs, but expressly for the purpose of encapsulating data name definitions within data dictionaries. No reference to these save frames is envisaged, and the save frame reference code permitted in STAR is not used. This means that unquoted character strings commencing with the $ character may not be interpreted as save frame codes in CIF. Use of such unquoted character strings is reserved to guard against subsequent relaxation of this constraint.

Exclusion of global_ blocks

33. In the full STAR specification, blocks of data headed by the special case-insensitive word global_ are permitted before normal data blocks. They contain data names and associated values which are inherited in subsequent data blocks; the scope of a value extends from its point of declaration in a global block to the end of the file. Because rearrangements of the order of data blocks and concatenation of data blocks from different files are commonplace operations in many CIF applications, and because of the difficulty in properly tracking and implementing values implied by global blocks, use of the global_ feature of STAR is expressly forbidden at this revision. To guard against its future introduction, the special case-insensitive word global_ remains reserved in CIF.

Version identification

34. As an archival file format, the CIF specification is expected to change infrequently. Revised specifications will be issued to accompany each substantial modification. A CIF may be considered compliant against the most recent version for which in practice it satisfies all syntactic and content rules as detailed in the formal specification document. However, to signal the version against which compliance was claimed at the time of creation, or to signal the file type and version to applications (such as operating-system utilities), it is recommended that a CIF begin with a structured comment that identifies the version of CIF used. For CIFs compliant with the current specification, the first 11 bytes of the file should be the string

#\#CIF_1.1

Appendix A: A formal grammar for CIF

Summary

Explanation of the Formal Syntax

39. In creating a parser for CIF, the normal process is to first perform a "lexical scan" to identify "tokens" in the CIF. A "token" is a grammatical unit, such as a special character, or a tag or a value, or some major grammatical subunit. In the course of a lexical scan, the input stream is reduced to manageable pieces, so that the rest of the parsing may be done more efficiently. The convention followed in this document is to mark the "non-terminal" tokens that are built up out of actual strings of characters or which do not have an immediate representation as printable characters by angle brackets, <>, and to indicate the tokens that are actual strings of characters as quoted strings of characters.

40. The precise division between a lexical scan and a full parse is a matter of convenience. A suggested division is presented. Before getting to that point, however, there are some highly machine-dependent matters that need to be resolved. There must be a clear understanding of the character set to be used, and of how files and lines begin and end. The character set will be specified in terms of printable characters and a few control characters from the venerable 7-bit ASCII (the US national variant of the ISO character set). In addition we will need some means of specifying the end of a line.

41. The character set in CIF is restricted to the ASCII control characters <HT> (horizontal tab, position 09 in the ASCII character set), <NL> (new line, position 10 in the ASCII character set, also named <LF>), and <CR> (carriage return, position 13 in the ASCII character set), and the printable characters in positions 32-126 of the ASCII characters set. These are the characters permitted by STAR with the exception of VT (vertical tab, position 11 in the ASCII character set) and FF (form feed, position 12 in the ASCII character set). In general it is poor practice to use characters which are not common to all national variants of the ISO character set. On systems or in programming languages which do not "work in ASCII", the characters themselves may have different numeric values and in some cases there is no access to all the control characters.

42. The <eol> token stands for the system-dependent end-of-line.

 <eol> ::= { <LF> |

<CR><LF> | <CR> } 

43. In order to write the grammar, we need a way to refer to the single-quote characters which we use both to quote within the syntax and to quote within a CIF. To avoid system dependent confusion, we define the following special tokens:

44. There are CIF specifications not definable directly in a context-free BNF. Restrictions in record and dataname lengths, and the parsing of text fields and quoted character strings are best handled in the initial lexical scan. A pure BNF can then be used to parse the tokenized input stream.

Lexical tokens

<Comments>               ::= { '#' {<AnyPrintChar>}* <eol>}+
<TokenizedComments>      ::= { <SP> | <HT> | <eol> |}+ <Comments>

<WhiteSpace>            ::= { <SP> | <HT> | <eol> | <TokenizedComments> }+ 

<NonBlankChar>          ::= <ordinary_char> | <double_quote> | '#' | '$'
      | <single_quote> | '_' | ';' | '[' | ']' 

<AnyPrintChar>          ::= <ordinary_char> | <double_quote> | '#' | '$' 
      | <single_quote> | '_' | <SP> | <HT> | ';' | '[' | ']' 

<TextLeadChar>          ::= <ordinary_char> | <double_quote> | '#' | '$' 
      | <single_quote> | '_' | <SP> | <HT> | '[' | ']' 

<OrdinaryChar>          ::= '!' | '%' | '&' | '(' | ')' | '*' | '+' | ',' | '-' | '.' 
      | '/' | '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' 
      | '9' | ':' | '<' | '=' | '>' | '?' | '@' | 'A' | 'B' | 'C' 
      | 'D' | 'E' | 'F' | 'G' | 'H' | 'I' | 'J' | 'K' | 'L' | 'M' 
      | 'N' | 'O' | 'P' | 'Q' | 'R' | 'S' | 'T' | 'U' | 'V' | 'W' 
      | 'X' | 'Y' | 'Z' | '\' | '^' | '`' | 'a' | 'b' | 'c' | 'd'
      | 'e' | 'f' | 'g' | 'h' | 'i' | 'j' | 'k' | 'l' | 'm' | 'n'
      | 'o' | 'p' | 'q' | 'r' | 's' | 't' | 'u' | 'v' | 'w' | 'x'
      | 'y' | 'z' | '{' | '|' | '}' | '~' |

<DATA_>                 ::= {'d'|'D'} {'a'|'A'} {'t'|'T'} {'a'|'A'} '_'

<LOOP_>                 ::= {'l'|'L'} {'o'|'O'} {'o'|'O'} {'p'|'P'} '_'

<SAVE_>                 ::= {'s'|'S'} {'a'|'A'} {'v'|'V'} {'e'|'E'} '_'

<STOP_>                 ::= {'s'|'S'} {'t'|'T'} {'o'|'O'} {'p'|'P'} '_'

<GLOBAL_>               ::= 
      {'g'|'G'} {'l'|'L'} {'o'|'O'} {'b'|'B'} {'a'|'A'} {'l'|'L'} '_'

<SingleQuotedString> <WhiteSpace> ::= <single_quote>{<AnyPrintChar>}* 
      <single_quote> <WhiteSpace>

<DoubleQuotedString> <WhiteSpace> ::= <double_quote> {<AnyPrintChar>}* 
      <double_quote> <WhiteSpace>
<TextField>             ::= { <SemiColonTextField>  }

<eol><SemiColonTextField>         ::= <eol>';' { {<AnyPrintChar>}* <eol>
            {{<TextLeadChar> {<AnyPrintChar>}*}? <eol>}*
            } ';'

<BracketTextField>      ::= '[' { <NonBracketChar> | <BracketTextField>}* ']'

<Tag>                   ::= '_'{ <NonBlankChar>}+
<Value>                 ::= { '.' | '?' | <Numeric> | <CharString> | <TextField> }

<Numeric>               ::= { <Number> | <Number> '(' <UnsignedInteger> ')' }
<Number>                ::= { <Integer> | <Float> }

<Integer>               ::= { '+' | '-' }? <UnsignedInteger>
<Exponent>              ::= { {'e' | 'E' } | {'e' | 'E' } { '+' | '- ' } } <UnsignedInteger>
<UnsignedInteger>       ::= { <Digit> }+
<Digit>                 ::= { '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' }

<Float>                 ::= { <Integer><Exponent> |
                                { {'+'|'-'} ? 
                                  { {<Digit>} * '.' <UnsignedInteger> } |
                                    { <Digit>} + '.' }
                                  }
                                  {<Exponent>} ?
                                }
                             }
<CharString>            ::= <UnquotedString> | <SingleQuotedString> | <DoubleQuotedString>

<eol><UnquotedString>    ::= <eol><OrdinaryChar> {<NonBlankChar>}*
<noteol><UnquotedString> ::= <noteol>{<OrdinaryChar>|';'} {<NonBlankChar>}*

CIF grammar

<CIF>                   ::=
      <Comments>? <WhiteSpace>? 
      { <DataBlock> { <WhiteSpace> <DataBlock> }* { <WhiteSpace> }? }?

<DataBlock>             ::=
      <DataBlockHeading> {<WhiteSpace> { <DataItems> | <SaveFrame> } }* 

<DataBlockHeading>      ::= <DATA_> { <NonBlankChar> }+

<SaveFrame>             ::= <SaveFrameHeading> {<WhiteSpace> <DataItems>}+
      <WhiteSpace> <SAVE_>

<SaveFrameHeading>      ::= <SAVE_> { <NonBlankChar> }+

<DataItems>             ::= <Tag> <WhiteSpace> <Value> |
         <LoopHeader> <LoopBody>

<LoopHeader>           ::= <LOOP_> { <WhiteSpace> <Tag> }+
<LoopBody>             ::= <Value> { <WhiteSpace> <Value> }*

References

• Hall, S. R. (1991). "The STAR File: A New Format for Electronic Data Transfer and Archiving", J. Chem. Inform. Comp. Sci., 31, 326-333.
• Hall, S.R. & Spadaccini, N. (1994). "The STAR File: Detailed Specifications," J. Chem. Info. Comp. Sci., 34, 505-508.