python-magic on GCP App Engine - failing to deploy

0 votes

I am working on Python app that will be deployed to GCP App Engine.
I tried to use python-magic functionality.
I've added import magic to my code and python-magic to the requirements.txt file.
But when I try to deploy code to App Engine it fails with the following error:

[2020-02-03 15:35:23 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/home/vmagent/app/main.py", line 11, in <module>
    import magic
  File "/env/lib/python3.6/site-packages/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation
[2020-02-03 15:35:23 +0000] [8] [INFO] Worker exiting (pid: 8)
[2020-02-03 15:35:23 +0000] [1] [INFO] Shutting down: Master
[2020-02-03 15:35:23 +0000] [1] [INFO] Reason: Worker failed to boot.

What I am missing to setup to make this work?


EDIT: sharing minimal setup to reproduce this error

main.py

# -*- coding: utf-8 -*-

import json
import base64
import magic

import logging
logging.basicConfig(level=logging.DEBUG)

from flask import Flask, request, abort

app = Flask(__name__)

@app.route('/parse', methods=['POST'])
def parse():

    try:
        #input should be a b64 encoded byte string
        input = request.get_json()['input']
    except Exception as e:
        abort(400, 'An error occured while reading in parameters: {}'.format(str(e)))

    decoded_input_data = base64.decodebytes(input)
    mime = magic.Magic(mime=True)
    file_ext = mime.from_buffer(decoded_input_data)

    returnData = {}
    returnData['Status'] = 200
    returnData['file_ext'] = file_ext

    return json.dumps(returnData), 200

@app.errorhandler(Exception)
def error(e):
    logging.exception(str(e))
    return json.dumps({'Status': e.code,
                       'Message': e.description}), e.code

if __name__ == '__main__':
    # This is used when running locally. Gunicorn is used to run the
    # application on Google App Engine. See entrypoint in app.yaml.
    app.run(host='127.0.0.1', port=8080, debug=True)

requirements.txt

Flask==1.0.2
gunicorn==19.9.0
google-cloud-vision==0.38.0
google-cloud-bigquery==1.11.2
google-cloud-storage==1.15.0
requests-toolbelt==0.9.1
protobuf==3.6.0
python-magic==0.4.15

app.yaml

runtime: python
env: flex
service: myservicename
entrypoint: gunicorn -b :$PORT main:app --timeout 240 --limit-request-line 0

runtime_config:
  python_version: 3
Apr 6, 2022 in GCP by Rahul
• 3,380 points
1,512 views

1 answer to this question.

0 votes

If you want to use the python-magic library you can follow the next procedure:

App Engine Standard

  1. Use the venv module to create an isolated Python environment in a directory external to your project and activate it with:
python3 -m venv env
source env/bin/activate
  1. Install the python-magic library with pip by running:
pip install python-magic
  1. Add the library to your requirements.txt file. For example, if you are running the Quickstart from the Official documentation your requirements.txt file should look like this:
Flask==1.1.1
python-magic==0.4.15
  1. Make the import of the library on the relevant module of your application's code. For example, if you are running the Quickstart from the Official documentation the relevant section of your main.py file should look like this:
...
# [START gae_python37_app]
from flask import Flask
import magic

# If `entrypoint` is not defined in app.yaml, App Engine will look for an app
# called `app` in `main.py`.
app = Flask(__name__)
...

App Engine Flexible The libmagic C library is not pre-installed on flex environment (here is the list of the installed libraries). You'll therefore need to create a custom runtime.

Changing your app.yaml to the following:

runtime: custom
env: flex

And adding the corresponding Dockerfile to your project's directory. For example:

FROM python:3.7
WORKDIR /app
COPY . /app
RUN apt-get update &&\
    apt-get install -y libmagic-dev
RUN pip install -r requirements.txt
EXPOSE 8080
CMD ["gunicorn", "main:app", "-b", ":8080", "--timeout", "300"]

should suffice to deploy the application successfully.

Notice that the Dockerfile simply installs the libmagic C library, afterwards installs your requirements.txt and finally runs gunicorn on the port required for App Engine. But feel free to adapt it to your needs.

answered Apr 6, 2022 by Korak
• 5,820 points

Related Questions In GCP

0 votes
1 answer

GCP App Engine Access to GCloud Storage without 'sharing publicly'

I believe that https://cloud.google.com/storage/docs/access-control/create-signed-urls-program will work for ...READ MORE

answered Nov 10, 2022 in GCP by Ashwini
• 5,430 points
796 views
+1 vote
2 answers

How to open a port on google compute engine

You need to set the firewall rule ...READ MORE

answered Jul 10, 2019 in GCP by K sagar
3,938 views
0 votes
2 answers

How to install gcp in Python?

If you're accessing BigQuery in python, you ...READ MORE

answered Aug 20, 2018 in GCP by Priyaj
• 58,100 points
1,684 views
+3 votes
1 answer

GCP Error: "Unable to connect on port 22"

You may see this error under the ...READ MORE

answered Sep 24, 2019 in GCP by Sirajul
• 59,230 points
7,977 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Python script, query to GCP postgresql db from local machine?

Using the Cloud SQL Python connector you ...READ MORE

answered Mar 8, 2022 in GCP by Korak
• 5,820 points
2,094 views
0 votes
1 answer

Why do I need to use a security solution like Centrify on GCP?

The main benefit is that access can ...READ MORE

answered Mar 17, 2022 in GCP by Korak
• 5,820 points
488 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP