externalise task def

To extend further more from previous example to show that you can use ref tag to externalise detailed task to separate files, in such a way it is much tidy and clear of your main skeleton of your program and focus what you want to achieve in high level

Relavant

Externalise tasks

references

c0081_task_compose_required_object.yml reference

    
    flow:
          -
            func: cmd
            vars:
              school_name: sydney grammar
              school_address: 1 fox road, sydney, nsw 2000
              schoo_principals:
                - peter
                - tom
                - jane
              schoo_ranking: No 5
            dvars:
              - name: school_string
                desc: use default expand == 1
                value: '{{.complex_data_structure_template}}'
            do:
              - name: print
                cmd: '{{.school_string}}'
    
          -
            func: cmd
            vars:
              school_name: sydney grammar
              school_address: 1 fox road, sydney, nsw 2000
              schoo_principals:
                - peter
                - tom
                - jane
              schoo_ranking: No 5
            dvars:
              - name: school_yml
                desc: use dynamic expand == 2, so that the template will be rendered
                value: '{{.complex_data_structure_template}}'
                expand: 2
                flags: [reg]
            do:
              - name: print
                cmd: '{{.school_yml}}'
    
          -
            func: cmd
            desc: |
              show global school_yml var
              convert the yml to object so that it could be used in template later
            dvars:
              - name: school_details
                desc: use dynamic expand == 2, so that the template will be rendered
                value: '{{.school_yml}}'
                flags: [vvv, toObj, reg]
            do:
              - name: print
                cmd: '{{.school_yml}}'
              - name: printObj
                cmd: 'school_details'
              - name: print
                cmd: '{{.school_details}}'
              - name: printObj
                cmd: 'school_details_object'
    
    

c0081_task_render.yml reference

    
    flow:
          - func: shell
            do:
              - echo "?intest ->  {{.intest}}"
              - pwd
            reg: register_render_root
    
          - func: cmd
            do:
              - name: print
                cmd: 'intest is: [{{.intest}}]'
    
              - name: reg
                cmd:
                  name: correct_working_dir
                  desc: the value of .intest is string but not bool so you can not simple use if .intest for condition
                  value: >-
                    {{if eq .intest "true" }}{{.register_render_root.Output}}{{else}}{{.register_render_root.Output}}{{end}}
    
          - func: cmd
            do:
              - name: print
                cmd: 'root dir is: [{{.register_render_root.Output}}]'
    
              - name: print
                cmd: 'correct working dir is: [{{.correct_working_dir}}]'
    
          -
            func: cmd
            desc: render final result using template
            do:
              - name: template
                desc: render the template file using above dynamic variable from defined var
                cmd:
                  src: '{{.correct_working_dir}}/tests/functests/d0079.template'
                  dest: /tmp/myschool.txt
                  datakey: "school_details_object"
    
          - func: cmd
            do:
              - name: readFile
                desc: read content of a file and register it to a var
                cmd:
                  filename: myschool.txt
                  dir: /tmp
                  reg: my_school
    
              -
                name: print
                cmd: "{{.my_school}}"
    

Demo

source

Main task yaml file
    docs:
      mddoc: |
        extend c0080 to use composible main tasks using externised flow configuration
         to make the whole task and flow clearer and modular
    vars:
      complex_data_structure_template: |
        school:
          name: '{{.school_name}}'
          address: '{{.school_address}}'
        principals:
        {{- range $_, $p :=.schoo_principals }}
          - {{$p}}
        {{- end}}
        ranking: '{{.schoo_ranking}}'
    dvars:
    - name: intest
      value: '{{ env "GOTEST" |default "false" }}'
    tasks:
    - name: task
      task:
      - func: call
        do:
        - compose_required_object
        - render
    - name: compose_required_object
      desc: prepare the data object for rendering
      ref: c0081_task_compose_required_object.yml
    - name: render
      desc: render data
      ref: c0081_task_render.yml
    
Main log file
    loading [Config]:  ./tests/functests/upconfig.yml
    Main config:
                 Version -> 1.0.0
                  RefDir -> ./tests/functests
                 WorkDir -> cwd
              AbsWorkDir -> /up_project/up
                TaskFile -> c0081
                 Verbose -> vvv
              ModuleName -> self
               ShellType -> /bin/sh
           MaxCallLayers -> 8
                 Timeout -> 3600000
     MaxModuelCallLayers -> 256
               EntryTask -> task
      ModRepoUsernameRef -> 
      ModRepoPasswordRef -> 
    work dir: /up_project/up
    -exec task: task
    loading [Task]:  ./tests/functests/c0081
    module: [self], instance id: [dev], exec profile: []
    profile -  envVars:
    
    (*core.Cache)({
    })
    
    loading [flow ref]:  ./tests/functests/c0081_task_compose_required_object.yml
    loading [flow ref]:  ./tests/functests/c0081_task_render.yml
    Task1: [task ==> task:  ]
    -Step1:
    self: final context exec vars:
    
    (*core.Cache)({
      "intest": "false",
      "up_runtime_task_layer_number": 0,
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n"
    })
    
    =Task2: [task ==> compose_required_object: prepare the data object for rendering ]
    --Step1:
    self: final context exec vars:
    
    (*core.Cache)({
      "school_address": "1 fox road, sydney, nsw 2000",
      "schoo_principals": {
        "peter",
        "tom",
        "jane"
      },
      "schoo_ranking": "No 5",
      "up_runtime_task_layer_number": 1,
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "intest": "false",
      "school_string": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "school_name": "sydney grammar"
    })
    
    ~~SubStep1: [print:  ]
    school:
      name: '{{.school_name}}'
      address: '{{.school_address}}'
    principals:
    {{- range $_, $p :=.schoo_principals }}
      - {{$p}}
    {{- end}}
    ranking: '{{.schoo_ranking}}'
    
    --Step2:
    self: final context exec vars:
    
    (*core.Cache)({
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_name": "sydney grammar",
      "school_address": "1 fox road, sydney, nsw 2000",
      "schoo_principals": {
        "peter",
        "tom",
        "jane"
      },
      "schoo_ranking": "No 5",
      "up_runtime_task_layer_number": 1,
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "intest": "false"
    })
    
    ~~SubStep1: [print:  ]
    school:
      name: 'sydney grammar'
      address: '1 fox road, sydney, nsw 2000'
    principals:
      - peter
      - tom
      - jane
    ranking: 'No 5'
    
    --Step3: [
    show global school_yml var
    convert the yml to object so that it could be used in template later
    ]
    dvar> school_details:
    "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n"
    
    -
    school:
      name: 'sydney grammar'
      address: '1 fox road, sydney, nsw 2000'
    principals:
      - peter
      - tom
      - jane
    ranking: 'No 5'
    
    dvar[object]> school_details_object:
    {
      "school": {
        "name": "sydney grammar",
        "address": "1 fox road, sydney, nsw 2000"
      },
      "principals": {
        "peter",
        "tom",
        "jane"
      },
      "ranking": "No 5"
    }
    
    self: final context exec vars:
    
    (*core.Cache)({
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "intest": "false",
      "up_runtime_task_layer_number": 1,
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_details_object": {
        "school": {
          "name": "sydney grammar",
          "address": "1 fox road, sydney, nsw 2000"
        },
        "principals": {
          "peter",
          "tom",
          "jane"
        },
        "ranking": "No 5"
      },
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n"
    })
    
    ~~SubStep1: [print:  ]
    school:
      name: 'sydney grammar'
      address: '1 fox road, sydney, nsw 2000'
    principals:
      - peter
      - tom
      - jane
    ranking: 'No 5'
    
    ~~SubStep2: [printObj:  ]
    object:
     school_details: "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n"
    
    ~~SubStep3: [print:  ]
    school:
      name: 'sydney grammar'
      address: '1 fox road, sydney, nsw 2000'
    principals:
      - peter
      - tom
      - jane
    ranking: 'No 5'
    
    ~~SubStep4: [printObj:  ]
    object:
     school_details_object: {
      "school": {
        "name": "sydney grammar",
        "address": "1 fox road, sydney, nsw 2000"
      },
      "principals": {
        "peter",
        "tom",
        "jane"
      },
      "ranking": "No 5"
    }
    
    =Task3: [task ==> render: render data ]
    --Step1:
    self: final context exec vars:
    
    (*core.Cache)({
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_details_object": {
        "ranking": "No 5",
        "school": {
          "name": "sydney grammar",
          "address": "1 fox road, sydney, nsw 2000"
        },
        "principals": {
          "peter",
          "tom",
          "jane"
        }
      },
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "intest": "false",
      "up_runtime_task_layer_number": 1
    })
    
    cmd( 1):
    echo "?intest ->  {{.intest}}"
    
    -
    ?intest ->  false
    
    -
     .. ok
    cmd( 2):
    pwd
    
    -
    /up_project/up
    
    -
     .. ok
    . ok
    --Step2:
    self: final context exec vars:
    
    (*core.Cache)({
      "last_result": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "intest": "false",
      "up_runtime_task_layer_number": 1,
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_details_object": {
        "ranking": "No 5",
        "school": {
          "address": "1 fox road, sydney, nsw 2000",
          "name": "sydney grammar"
        },
        "principals": {
          "peter",
          "tom",
          "jane"
        }
      },
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "register_render_root": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      })
    })
    
    ~~SubStep1: [print:  ]
    intest is: [false]
    ~~SubStep2: [reg:  ]
    --Step3:
    self: final context exec vars:
    
    (*core.Cache)({
      "correct_working_dir": "/up_project/up",
      "up_runtime_task_layer_number": 1,
      "intest": "false",
      "school_details_object": {
        "ranking": "No 5",
        "school": {
          "name": "sydney grammar",
          "address": "1 fox road, sydney, nsw 2000"
        },
        "principals": {
          "peter",
          "tom",
          "jane"
        }
      },
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "last_result": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "register_render_root": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      })
    })
    
    ~~SubStep1: [print:  ]
    root dir is: [/up_project/up]
    ~~SubStep2: [print:  ]
    correct working dir is: [/up_project/up]
    --Step4: [: render final result using template ]
    self: final context exec vars:
    
    (*core.Cache)({
      "register_render_root": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "last_result": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "correct_working_dir": "/up_project/up",
      "intest": "false",
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "up_runtime_task_layer_number": 1,
      "school_details_object": {
        "school": {
          "name": "sydney grammar",
          "address": "1 fox road, sydney, nsw 2000"
        },
        "principals": {
          "peter",
          "tom",
          "jane"
        },
        "ranking": "No 5"
      }
    })
    
    ~~SubStep1: [template: render the template file using above dynamic variable from defined var ]
    --Step5:
    self: final context exec vars:
    
    (*core.Cache)({
      "register_render_root": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "complex_data_structure_template": "school:\n  name: '{{.school_name}}'\n  address: '{{.school_address}}'\nprincipals:\n{{- range $_, $p :=.schoo_principals }}\n  - {{$p}}\n{{- end}}\nranking: '{{.schoo_ranking}}'\n",
      "school_yml": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "intest": "false",
      "school_details": "school:\n  name: 'sydney grammar'\n  address: '1 fox road, sydney, nsw 2000'\nprincipals:\n  - peter\n  - tom\n  - jane\nranking: 'No 5'\n",
      "school_details_object": {
        "principals": {
          "peter",
          "tom",
          "jane"
        },
        "ranking": "No 5",
        "school": {
          "address": "1 fox road, sydney, nsw 2000",
          "name": "sydney grammar"
        }
      },
      "last_result": (*utils.ExecResult)({
        Cmd: "pwd",
        Code: 0,
        Output: "/up_project/up",
        ErrMsg: ""
      }),
      "correct_working_dir": "/up_project/up",
      "up_runtime_task_layer_number": 1
    })
    
    ~~SubStep1: [readFile: read content of a file and register it to a var ]
    ~~SubStep2: [print:  ]
    
    
    My school name is: sydney grammar, it is located
    at 1 fox road, sydney, nsw 2000
    
    
    We have got  peter,  tom,  jane,  as our principals
    
    Our school ranking last year is No 5
    
    
Logs with different verbose level
Raw logs with different verbose level