Data retrieval and Querying Elasticsearch with Python
- or -
Post a project like this1913
€10/hr(approx. $11/hr)
- Posted:
- Proposals: 4
- Remote
- #2268701
- Awarded
Description
Experience Level: Entry
Estimated project duration: 1 day or less
Hi there,
I am trying to get data from Elasticsearch using python. I was able to connect to the data source using python.
Connect To Elasticsearch
from elasticsearch import Elasticsearch
try:
es = Elasticsearch(
['https:/movies-elk-prod-elastic-ssl.pac.com'],
http_auth=('xxxxx', 'xxxx'),
port=10202,
)
print ("Connected", es.info())
except Exception as ex:
print ("Error:", ex)
es.info(pretty=True)
The Input Data
{
"_index": "movies",
"_type": "movie",
"_id": "4444169",
"_score": null,
"_source": {
"actor": [
"Josh Duhamel",
"Megan Fox",
"Shia LaBeouf"
],
"director": [
"Michael Bay"
],
"full_text": "When Sam Witwicky learns the truth about the ancient origins of the Transformers, he must accept his fate and merge with Optimus Prime and Bumblebee in their epic battle against the Decepticons, who are stronger back than ever and plan to destroy our world.!",
"title": "Transformers: Revenge of the Fallen",
"type": "movie"
},
"sort": [
1544310000000,
4.05
]
},
{
"_index": "movies",
"_type": "movie",
"_id": "4051",
"_score": null,
"_source": {
"actor": [
"Josh Duhamel",
"Shia LaBeouf",
"Patrick Dempsey"
],
"director": [
"Michael Bay"
],
"full_text": "A mysterious event from the Earth's past threatens to unleash such a devastating war that the Transformers can not possibly save the planet on its own. Sam Witwicky and the Autobots fight the darkness to defend our world against the devastating evil of the Decepticons.",
"title": "Transformers: Dark of the Moon",
"type": "movie"
},
"sort": [
1544310000000,
4.03949
]
},
Next, I want to write a python function which queries the Elasticsearch based on actor e.g suppose if I write Josh Duhamel. It should give me all the movies containing Josh Duhamel.
As a next step, I want to convert the data into a python data frame. I tried a few things based on functions I have found this but it's not working for me (Ps. I am new to Elasticsearch as well as python -:))
def search(uri, term):
"""Simple Elasticsearch Query"""
query = json.dumps({
"query": {
"match": {
"content": term
}
}
})
response = requests.get(uri, data=query)
results = json.loads(response.text)
return results
def format_results(results):
"""Print results nicely:
doc_id) content
"""
data = [doc for doc in results['hits']['hits']]
for doc in data:
print("%s) %s" % (doc['_id'], doc['_source']['content'])
def create_doc(uri, doc_data={}):
"""Create new document."""
query = json.dumps(doc_data)
response = requests.post(uri, data=query)
print(response)
from elasticsearch import Elasticsearch
es = Elasticsearch()
res = es.search(index="test", doc_type="articles", body={"query": {"match": {"content": "fox"}}})
print("%d documents found" % res['hits']['total'])
for doc in res['hits']['hits']:
print("%s) %s" % (doc['_id'], doc['_source']['content']))
Thanks in advance
I am trying to get data from Elasticsearch using python. I was able to connect to the data source using python.
Connect To Elasticsearch
from elasticsearch import Elasticsearch
try:
es = Elasticsearch(
['https:/movies-elk-prod-elastic-ssl.pac.com'],
http_auth=('xxxxx', 'xxxx'),
port=10202,
)
print ("Connected", es.info())
except Exception as ex:
print ("Error:", ex)
es.info(pretty=True)
The Input Data
{
"_index": "movies",
"_type": "movie",
"_id": "4444169",
"_score": null,
"_source": {
"actor": [
"Josh Duhamel",
"Megan Fox",
"Shia LaBeouf"
],
"director": [
"Michael Bay"
],
"full_text": "When Sam Witwicky learns the truth about the ancient origins of the Transformers, he must accept his fate and merge with Optimus Prime and Bumblebee in their epic battle against the Decepticons, who are stronger back than ever and plan to destroy our world.!",
"title": "Transformers: Revenge of the Fallen",
"type": "movie"
},
"sort": [
1544310000000,
4.05
]
},
{
"_index": "movies",
"_type": "movie",
"_id": "4051",
"_score": null,
"_source": {
"actor": [
"Josh Duhamel",
"Shia LaBeouf",
"Patrick Dempsey"
],
"director": [
"Michael Bay"
],
"full_text": "A mysterious event from the Earth's past threatens to unleash such a devastating war that the Transformers can not possibly save the planet on its own. Sam Witwicky and the Autobots fight the darkness to defend our world against the devastating evil of the Decepticons.",
"title": "Transformers: Dark of the Moon",
"type": "movie"
},
"sort": [
1544310000000,
4.03949
]
},
Next, I want to write a python function which queries the Elasticsearch based on actor e.g suppose if I write Josh Duhamel. It should give me all the movies containing Josh Duhamel.
As a next step, I want to convert the data into a python data frame. I tried a few things based on functions I have found this but it's not working for me (Ps. I am new to Elasticsearch as well as python -:))
def search(uri, term):
"""Simple Elasticsearch Query"""
query = json.dumps({
"query": {
"match": {
"content": term
}
}
})
response = requests.get(uri, data=query)
results = json.loads(response.text)
return results
def format_results(results):
"""Print results nicely:
doc_id) content
"""
data = [doc for doc in results['hits']['hits']]
for doc in data:
print("%s) %s" % (doc['_id'], doc['_source']['content'])
def create_doc(uri, doc_data={}):
"""Create new document."""
query = json.dumps(doc_data)
response = requests.post(uri, data=query)
print(response)
from elasticsearch import Elasticsearch
es = Elasticsearch()
res = es.search(index="test", doc_type="articles", body={"query": {"match": {"content": "fox"}}})
print("%d documents found" % res['hits']['total'])
for doc in res['hits']['hits']:
print("%s) %s" % (doc['_id'], doc['_source']['content']))
Thanks in advance
South D.
100% (29)Projects Completed
22
Freelancers worked with
18
Projects awarded
42%
Last project
25 Jan 2023
Netherlands
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
There are no clarification messages.
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies