class TaskJuggler::TextParser::Pattern

This class models the most crutial elements of a syntax description - the pattern. A TextParserPattern primarily consists of a set of tokens. Tokens are Strings where the first character determines the type of the token. There are 4 known types.

Terminal token: In the syntax declaration the terminal token is prefixed by an underscore. Terminal tokens are terminal symbols of the syntax tree. They just represent themselves.

Variable token: The variable token describes values of a certain class such as strings or numbers. In the syntax declaration the token is prefixed by a dollar sign and the text of the token specifies the variable type. See ProjectFileParser for a complete list of variable types.

Reference token: The reference token specifies a reference to another parser rule. In the syntax declaration the token is prefixed by a bang and the text matches the name of the rule. See TextParserRule for details.

End token: The . token marks the expected end of the input stream.

In addition to the pure syntax tree information the pattern also holds documentary information about the pattern.



Public Class Methods

new(tokens, function = nil) click to toggle source

Create a new Pattern object. tokens must be an Array of String objects that describe the Pattern. function can be a reference to a method that should be called when the Pattern was recognized by the parser.

# File lib/taskjuggler/TextParser/Pattern.rb, line 48
def initialize(tokens, function = nil)
  # A unique name for the pattern that is used in the documentation.
  @keyword = nil
  # Initialize pattern doc as empty.
  @doc = nil
  # A list of TokenDoc elements that describe the meaning of variable
  # tokens. The order of the tokens and entries in the Array must correlate.
  @args = []
  # The syntax can evolve over time. The support level specifies which
  # level of support this pattern hast. Possible values are :experimental,
  # :beta, :supported, :deprecated, :removed
  @supportLevel = :supported
  # A list of references to other patterns that are related to this pattern.
  @seeAlso = []
  # A reference to a file under test/TestSuite/Syntax/Correct and a tag
  # within that file. This identifies example TJP code to be included with
  # the reference manual.
  @exampleFile = nil
  @exampleTag = nil

  @tokens = []
  tokens.each do |token|
    unless '!$_.'.include?(token[0])
      raise "Fatal Error: All pattern tokens must start with a type " +
            "identifier [!$_.]: #{tokens.join(', ')}"
    # For the syntax specification using a prefix character is more
    # convenient. But for further processing, we need to split the string
    # into two symbols. The prefix determines the token type, the rest is
    # the token name. There are 4 types of tokens:
    # :reference : a reference to another rule
    # :variable : a terminal symbol
    # :literal : a user defined string
    # :eof : marks the end of an input stream
    type = [ :reference, :variable, :literal, :eof ]['!$_.'.index(token[0])]
    # For literals we use a String to store the token content. For others,
    # a symbol is better suited.
    name = type == :literal ?
           token[1..-1] : (type == :eof ? '<END>' : token[1..-1].intern)
    # We favor an Array to store the 2 elements over a Hash for
    # performance reasons.
    @tokens << [ type, name ]
    # Initialize pattern argument descriptions as empty.
    @args << nil
  @function = function
  # In some cases we don't want to show all tokens in the syntax
  # documentation. This value specifies the index of the last shown token.
  @lastSyntaxToken = @tokens.length - 1

  @transitions = []

Public Instance Methods

[](i) click to toggle source

Conveniance function to access individual tokens by index.

# File lib/taskjuggler/TextParser/Pattern.rb, line 245
def [](i)
addTransitionsToState(states, rules, stateStack, sourceState, destRule, destIndex, loopBack) click to toggle source

Add the transitions to the State objects of this pattern. states is a Hash with all State objects. rules is a Hash with the Rule objects of the syntax. stateStack is an Array of State objects that have been traversed before reaching this pattern. sourceState is the State that the transition originates from. destRule, this pattern and destIndex describe the State the transition is leading to. loopBack is boolean flag, set to true when the transition describes a loop back to the start of the Rule.

# File lib/taskjuggler/TextParser/Pattern.rb, line 140
def addTransitionsToState(states, rules, stateStack, sourceState,
                          destRule, destIndex, loopBack)
  # If we hit a token in the pattern that is optional, we need to consider
  # the next token of the pattern as well.
  loop do
    if destIndex >= @tokens.length
      if sourceState.rule == destRule
        if destRule.repeatable
          # The transition leads us back to the start of the Rule. This
          # will generate transitions to the first token of all patterns
          # of this Rule.
          destRule.addTransitionsToState(states, rules, [], sourceState,
      # We've reached the end of the pattern. No more transitions to
      # consider.

    # The token descriptor tells us where the transition(s) need to go to.
    tokenType, tokenName = @tokens[destIndex]

    case tokenType
    when :reference
      # The descriptor references another rule.
      unless (refRule = rules[tokenName])
        raise "Unknown rule #{tokenName} referenced in rule #{}"
      # If we reference another rule from a pattern, we need to come back
      # to the pattern once we are done with the referenced rule. To be
      # able to come back, we collect a list of all the States that we
      # have passed during a reference resolution. This list forms a stack
      # that is popped during recude operations of the parser FSM.
      skippedState = states[[ destRule, self, destIndex ]]
      # Rules may reference themselves directly or indirectly. To avoid
      # endless recursions of this algorithm, we stop once we have
      # detected a recursion. We have already all necessary transitions
      # collected. The recursion will be unrolled in the parser FSM.
      unless stateStack.include?(skippedState)
        # Push the skipped state on the stateStack before recursing.
        refRule.addTransitionsToState(states, rules, stateStack,
                                      sourceState, loopBack)
        # Once we're done, remove the State from the stateStack again.

      # If the referenced rule is not optional, we have no further
      # transitions for this pattern at this destIndex.
      break unless refRule.optional?(rules)
      unless (destState = states[[ destRule, self, destIndex ]])
        raise "Destination state not found"
      # We've found a transition to a terminal token. Add the transition
      # to the source State.
      sourceState.addTransition(@tokens[destIndex], destState, stateStack,
      # Fixed tokens are never optional. There are no more transitions for
      # this pattern at this index.

    destIndex += 1
each() { |type, name| ... } click to toggle source

Iterator for tokens.

# File lib/taskjuggler/TextParser/Pattern.rb, line 250
def each
  @tokens.each { |type, name| yield(type, name) }
empty?() click to toggle source

Returns true of the pattern is empty.

# File lib/taskjuggler/TextParser/Pattern.rb, line 255
def empty?
generateStates(rule, rules) click to toggle source

Generate the state machine states for the pattern. rule is the Rule that the pattern belongs to. A list of generated State objects will be returned.

# File lib/taskjuggler/TextParser/Pattern.rb, line 104
def generateStates(rule, rules)
  # The last token of a pattern must always trigger a reduce operation.
  # But the the last tokens of a pattern describe fully optional syntax,
  # the last non-optional token and all following optional tokens must
  # trigger a reduce operation. Here we find the index of the first token
  # that must trigger a reduce operation.
  firstReduceableToken = @tokens.length - 1
  (@tokens.length - 2).downto(0).each do |i|
    if optionalToken(i + 1, rules)
      # If token i + 1 is optional, assume token i is the first one to
      # trigger a reduce.
      firstReduceableToken = i
      # token i + 1 is not optional, we found the first token to trigger
      # the reduce.

  states = []
  @tokens.length.times do |i|
    states << (state =, self, i))
    # Mark all states that are allowed to trigger a reduce operation.
    state.noReduce = false if i >= firstReduceableToken
length() click to toggle source

Returns the number of tokens in the pattern.

# File lib/taskjuggler/TextParser/Pattern.rb, line 260
def length
optional?(rules) click to toggle source

Return true if all tokens of the pattern are optional. If a token references a rule, this rule is followed for the check.

# File lib/taskjuggler/TextParser/Pattern.rb, line 266
def optional?(rules)
  @tokens.each do |type, name|
    if type == :literal || type == :variable
      return false
    elsif type == :reference
      if !rules[name].optional?(rules)
        return false
setArg(idx, doc) click to toggle source

Set the documentation text and for the idx-th variable.

# File lib/taskjuggler/TextParser/Pattern.rb, line 215
def setArg(idx, doc)
  @args[idx] = doc
setDoc(keyword, doc) click to toggle source

Set the keyword and documentation text for the pattern.

# File lib/taskjuggler/TextParser/Pattern.rb, line 209
def setDoc(keyword, doc)
  @keyword = keyword
  @doc = doc
setExample(file, tag) click to toggle source

Set the file and tag for the TJP code example.

# File lib/taskjuggler/TextParser/Pattern.rb, line 239
def setExample(file, tag)
  @exampleFile = file
  @exampleTag = tag
setLastSyntaxToken(idx) click to toggle source

Restrict the syntax documentation to the first idx tokens.

# File lib/taskjuggler/TextParser/Pattern.rb, line 220
def setLastSyntaxToken(idx)
  @lastSyntaxToken = idx
setSeeAlso(also) click to toggle source

Set the references to related patterns.

# File lib/taskjuggler/TextParser/Pattern.rb, line 234
def setSeeAlso(also)
  @seeAlso = also
setSupportLevel(level) click to toggle source

Specify the support level of this pattern.

# File lib/taskjuggler/TextParser/Pattern.rb, line 225
def setSupportLevel(level)
  unless [ :experimental, :beta, :supported, :deprecated,
           :removed ].include?(level)
    raise "Fatal Error: Unknown support level #{level}"
  @supportLevel = level
terminalSymbol?(i) click to toggle source

Returns true if the i-th token is a terminal symbol.

# File lib/taskjuggler/TextParser/Pattern.rb, line 280
def terminalSymbol?(i)
  @tokens[i][0] == :variable || @tokens[i][0] == :literal
terminalTokens(rules, index = 0) click to toggle source

Find recursively the first terminal token of this pattern. If an index is specified start the search at this n-th pattern token instead of the first. The return value is an Array of [ token, pattern ] tuple.

# File lib/taskjuggler/TextParser/Pattern.rb, line 287
def terminalTokens(rules, index = 0)
  type, name = @tokens[index]
  # Terminal token start with an underscore or dollar character.
  if type == :literal
    return [ [ name, self ] ]
  elsif type == :variable
    return []
  elsif type == :reference
    # We have to continue the search at this rule.
    rule = rules[name]
    # The rule may only have a single pattern. If not, then this pattern
    # has no terminal token.
    tts = []
    rule.patterns.each { |p| tts += p.terminalTokens(rules, 0) }
    return tts
    raise "Unexpected token #{type} #{name}"
to_s() click to toggle source

Generate a text form of the pattern. This is similar to the syntax in the original syntax description.

# File lib/taskjuggler/TextParser/Pattern.rb, line 389
def to_s
  str = ""
  @tokens.each do |type, name|
    case type
    when :reference
      str += "!#{name} "
    when :variable
      str += "$#{name } "
    when :literal
      str += "#{name} "
    when :eof
      str += ". "
      raise "Unknown type #{type}"

to_syntax(argDocs, rules, skip = 0) click to toggle source

Returns a string that expresses the elements of the pattern in an EBNF like fashion. The resolution of the pattern is done recursively. This is just the wrapper function that sets up the stack.

# File lib/taskjuggler/TextParser/Pattern.rb, line 310
def to_syntax(argDocs, rules, skip = 0)
  to_syntax_r({}, argDocs, rules, skip)
to_syntax_r(stack, argDocs, rules, skip) click to toggle source

Generate a syntax description for this pattern.

# File lib/taskjuggler/TextParser/Pattern.rb, line 315
def to_syntax_r(stack, argDocs, rules, skip)
  # If we find ourself on the stack we hit a recursive pattern. This is used
  # in repetitions.
  if stack[self]
    return '[, ... ]'

  # "Push" us on the stack.
  stack[self] = true

  str = ''
  first = true
  # Analyze the tokens of the pattern skipping the first 'skip' tokens.
  skip.upto(@lastSyntaxToken) do |i|
    type, name = @tokens[i]
    # If the first token is a _{ the pattern describes optional attributes.
    # They are represented by a standard idiom.
    if first
      first = false
      return '{ <attributes> }' if name == '{'
      # Separate the syntax elemens by a whitespace.
      str << ' '

    if @args[i]
      # The argument is documented in the syntax definition. We copy the
      # entry as we need to modify it.
      argDoc = @args[i].dup

      # A documented argument without a name is a terminal token. We use the
      # terminal symbol as name.
      if @args[i].name.nil?
        str << "#{name}" = name
        str << "<#{@args[i].name}>"
      addArgDoc(argDocs, argDoc)

      # Documented arguments don't have the type set yet. Use the token
      # value for that.
      if type == :variable
        argDoc.typeSpec = "<#{name}>"
      # Undocumented tokens are recursively expanded.
      case type
      when :literal
        # Literals are shown as such.
        str << name.to_s
      when :variable
        # Variables are enclosed by angle brackets.
        str << "<#{name}>"
      when :reference
        if rules[name].patterns.length == 1 &&
          str << '<' + rules[name].patterns[0].keyword + '>'
          # References are followed recursively.
          str << rules[name].to_syntax(stack, argDocs, rules, 0)
  # Remove us from the "stack" again.