Ansibleでgoogle-fluentdをインストールし、Stackdriver Loggingでログを確認する

やったこと

Ansibleでgoogle-fluentdをインストールして、GCPのStackdriver-Loggingでログの転送を確認する。 Ansibleで設定を行う。環境はUbuntu16.04で行った。

Ansibleのコード

■ Ansibleディレクトリ

roles/google-fluentd/
├── files
│   └── key.json
├── handlers
│   └── main.yml
├── tasks
│   └── main.yml
└── templates
    └── log-collect.conf.j2

■ tasks/main.yml

---
- name: Create /etc/google/auth
  file:
    path: /etc/google/auth
    state: directory
    mode: 0755
    owner: root
    group: root
  become: yes


- name: Copy credential
  copy:
    src: key.json
    dest: /etc/google/auth/application_default_credentials.json
    mode: 0400
    owner: root
    group: root
  become: yes


- name: Check if google-fluentd is installed
  command: dpkg-query -l google-fluentd
  register: gf_check
  check_mode: no
  ignore_errors: yes


- name: Download install-logging-agent.sh
  get_url:
    url: https://dl.google.com/cloudagents/install-logging-agent.sh
    dest: /usr/local/install-logging-agent.sh
    mode: 0744
    force: yes
  become: yes
  when: gf_check.stderr.find('no packages found') != -1


- name: Install install-logging-agent
  command: ./install-logging-agent.sh
  args:
    chdir: /usr/local
  notify: "Restart google-fluentd"
  become: yes
  when: gf_check.stderr.find('no packages found') != -1


- name: Install fluent plugin
  gem:
    name: "{{ item }}"
    executable: /opt/google-fluentd/embedded/bin/fluent-gem
    state: latest
    user_install: no
  with_items:
    - fluent-plugin-s3
    - fluent-plugin-systemd
    - fluent-plugin-forest
    - fluent-plugin-rewrite-tag-filter
    - fluent-plugin-multi-format-parser
  notify: "Restart google-fluentd"
  become: yes
  when: not ansible_check_mode


- name: Copy google-fluentd config
  template:
    src: log-collect.conf.j2
    dest: /etc/google-fluentd/config.d/syslog.conf
    mode: 0644
  notify: "Restart google-fluentd"
  become: yes


- name: Ensure start google-fluentd services
  systemd:
    state: started
    daemon_reload: yes
    enabled: yes
    name: "{{ item }}"
  with_items:
    - google-fluentd.service
  become: yes


- name: Remove unused conf files
  file:
    state: absent
    path: "/etc/google-fluentd/config.d/{{ item }}"
  with_items:
    - apache.conf
    - cassandra.conf
    - chef.conf
    - forward.conf
    - gitlab.conf
    - jenkins.conf
    - jetty.conf
    - joomla.conf
    - magento.conf
    - mediawiki.conf
    - memcached.conf
    - mongodb.conf
    - mysql.conf
    - postgresql.conf
    - puppet.conf
    - puppet-enterprise.conf
    - rabbitmq.conf
    - redis.conf
    - redmine.conf
    - salt.conf
    - solr.conf
    - sugarcrm.conf
    - syslog_endpoint.conf
    - tomcat.conf
    - zookeeper.conf
  become: yes

■ templates/log-collect.conf.j2

<source>
  @type http
  port 8080
  bind 0.0.0.0
</source>

<source>
  @type systemd
  path /run/log/journal
  tag journal
  <storage>
    @type local
    persistent false
    path /var/lib/google-fluentd/pos/journal.pos
  </storage>
  <entry>
    field_map {"MESSAGE": "message", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

■ handlers/main.yml

---
- name: "Restart google-fluentd"
  systemd:
    name: google-fluentd
    state: restarted
    daemon_reload: yes
  become: yes

■ files/key.json (ansible vaultで暗号化する)

$ cat key.json
$ANSIBLE_VAULT;1.1;AES256
39646166643161366438346136386264326330306231613961393732613531623864613934656633
6661343939373262336433626465326266386361326332380a326538323534636439376638343736
65333834633334663732353539386365356433353561646234303032653566623539353031616262
6137653166343339330a343464653062666432333638323534383338353939386238396262393537
・・・
・・・

key.json（暗号化前）に記載する情報を取得する

1. IAMと管理
2. サービスアカウント
3. サービスアカウントを作成
4. サービスアカウント名で stackdriver-logging など任意の名前をつける
5. 権限を loggingのログを書き込む権限を付与する
6. keyを作成するで、json形式でkey情報を取得する

gcp-logging-key

→ このファイルをansible-vaultで暗号化した。

Stackdriver Loggingで転送ログを確認する

gcp-logging-journal