mindol1004/spring-data-mongodb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DATAMONGO-1870 - Consider skip/limit on MongoOperations.remove(Query, Class).

Browse Source 
We now use _id lookup for remove operations that query with limit or skip parameters. This allows more fine grained control over documents removed.

Original pull request: #532.
Related pull request: #531.
This commit is contained in:
Christoph Strobl
2018-02-12 13:24:53 +01:00
committed by Mark Paluch
parent 4ebcac19bc
commit c873e49d71
8 changed files with 156 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
spring-data-mongodb/src/main/java/org/springframework/data/mongodb/core/MongoOperations.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2011-2017 the original author or authors.
* Copyright 2011-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -710,8 +710,8 @@ public interface MongoOperations extends FluentMongoOperations {
<T> T findById(Object id, Class<T> entityClass, String collectionName);
/**
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify<a/>
* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query}.
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify
* <a/>* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query}.
*
* @param query the {@link Query} class that specifies the {@link Criteria} used to find a record and also an optional
* fields specification. Must not be {@literal null}.
@@ -723,8 +723,8 @@ public interface MongoOperations extends FluentMongoOperations {
<T> T findAndModify(Query query, Update update, Class<T> entityClass);
/**
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify<a/>
* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query}.
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify
* <a/>* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query}.
*
* @param query the {@link Query} class that specifies the {@link Criteria} used to find a record and also an optional
* fields specification. Must not be {@literal null}.
@@ -737,8 +737,8 @@ public interface MongoOperations extends FluentMongoOperations {
<T> T findAndModify(Query query, Update update, Class<T> entityClass, String collectionName);
/**
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify<a/>
* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query} taking
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify
* <a/>* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query} taking
* {@link FindAndModifyOptions} into account.
*
* @param query the {@link Query} class that specifies the {@link Criteria} used to find a record and also an optional
@@ -754,8 +754,8 @@ public interface MongoOperations extends FluentMongoOperations {
<T> T findAndModify(Query query, Update update, FindAndModifyOptions options, Class<T> entityClass);
/**
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify<a/>
* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query} taking
* Triggers <a href="https://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/">findAndModify
* <a/>* to apply provided {@link Update} on documents matching {@link Criteria} of given {@link Query} taking
* {@link FindAndModifyOptions} into account.
*
* @param query the {@link Query} class that specifies the {@link Criteria} used to find a record and also an optional
@@ -1083,6 +1083,7 @@ public interface MongoOperations extends FluentMongoOperations {
* @param query the query document that specifies the criteria used to remove a record.
* @param entityClass class that determines the collection to use.
* @return the {@link DeleteResult} which lets you access the results of the previous delete.
* @throws IllegalArgumentException when {@literal query} or {@literal entityClass} is {@literal null}.
*/
DeleteResult remove(Query query, Class<?> entityClass);
@@ -1094,6 +1095,8 @@ public interface MongoOperations extends FluentMongoOperations {
* @param entityClass class of the pojo to be operated on. Can be {@literal null}.
* @param collectionName name of the collection where the objects will removed, must not be {@literal null} or empty.
* @return the {@link DeleteResult} which lets you access the results of the previous delete.
* @throws IllegalArgumentException when {@literal query}, {@literal entityClass} or {@literal collectionName} is
* {@literal null}.
*/
DeleteResult remove(Query query, Class<?> entityClass, String collectionName);
@@ -1106,6 +1109,7 @@ public interface MongoOperations extends FluentMongoOperations {
* @param query the query document that specifies the criteria used to remove a record.
* @param collectionName name of the collection where the objects will removed, must not be {@literal null} or empty.
* @return the {@link DeleteResult} which lets you access the results of the previous delete.
* @throws IllegalArgumentException when {@literal query} or {@literal collectionName} is {@literal null}.
*/
DeleteResult remove(Query query, String collectionName);

54
spring-data-mongodb/src/main/java/org/springframework/data/mongodb/core/MongoTemplate.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2010-2017 the original author or authors.
* Copyright 2010-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -18,26 +18,14 @@ package org.springframework.data.mongodb.core;
import static org.springframework.data.mongodb.core.query.Criteria.*;
import static org.springframework.data.mongodb.core.query.SerializationUtils.*;
import com.mongodb.*;
import lombok.AccessLevel;
import lombok.AllArgsConstructor;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.*;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Scanner;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.bson.Document;
@@ -128,6 +116,14 @@ import org.springframework.util.ObjectUtils;
import org.springframework.util.ResourceUtils;
import org.springframework.util.StringUtils;
import com.mongodb.Cursor;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.Mongo;
import com.mongodb.MongoClient;
import com.mongodb.MongoException;
import com.mongodb.ReadPreference;
import com.mongodb.WriteConcern;
import com.mongodb.client.AggregateIterable;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MapReduceIterable;
@@ -1587,13 +1583,11 @@ public class MongoTemplate implements MongoOperations, ApplicationContextAware,
protected <T> DeleteResult doRemove(final String collectionName, final Query query,
@Nullable final Class<T> entityClass) {
Assert.notNull(query, "Query must not be null!");
Assert.hasText(collectionName, "Collection name must not be null or empty!");
if (query == null) {
throw new InvalidDataAccessApiUsageException("Query passed in to remove can't be null!");
}
final Document queryObject = query.getQueryObject();
final MongoPersistentEntity<?> entity = getPersistentEntity(entityClass);
final Document queryObject = queryMapper.getMappedObject(query.getQueryObject(), entity);
return execute(collectionName, new CollectionCallback<DeleteResult>() {
@@ -1602,7 +1596,7 @@ public class MongoTemplate implements MongoOperations, ApplicationContextAware,
maybeEmitEvent(new BeforeDeleteEvent<T>(queryObject, entityClass, collectionName));
Document mappedQuery = queryMapper.getMappedObject(queryObject, entity);
Document removeQuery = queryObject;
DeleteOptions options = new DeleteOptions();
query.getCollation().map(Collation::toMongoCollation).ifPresent(options::collation);
@@ -1615,14 +1609,27 @@ public class MongoTemplate implements MongoOperations, ApplicationContextAware,
DeleteResult dr = null;
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Remove using query: {} in collection: {}.",
new Object[] { serializeToJsonSafely(mappedQuery), collectionName });
new Object[] { serializeToJsonSafely(removeQuery), collectionName });
}
if (query.getLimit() > 0 || query.getSkip() > 0) {
MongoCursor<Document> cursor = new QueryCursorPreparer(query, entityClass)
.prepare(collection.find(removeQuery).projection(new Document(ID_FIELD, 1))).iterator();
Set<Object> ids = new LinkedHashSet<>();
while (cursor.hasNext()) {
ids.add(cursor.next().get(ID_FIELD));
}
removeQuery = new Document(ID_FIELD, new Document("$in", ids));
}
if (writeConcernToUse == null) {
dr = collection.deleteMany(mappedQuery, options);
dr = collection.deleteMany(removeQuery, options);
} else {
dr = collection.withWriteConcern(writeConcernToUse).deleteMany(mappedQuery, options);
dr = collection.withWriteConcern(writeConcernToUse).deleteMany(removeQuery, options);
}
maybeEmitEvent(new AfterDeleteEvent<T>(queryObject, entityClass, collectionName));
@@ -3097,8 +3104,7 @@ public class MongoTemplate implements MongoOperations, ApplicationContextAware,
*/
Document aggregate(String collectionName, Aggregation aggregation, AggregationOperationContext context) {
Document command = prepareAggregationCommand(collectionName, aggregation,
context, batchSize);
Document command = prepareAggregationCommand(collectionName, aggregation, context, batchSize);
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Executing aggregation: {}", serializeToJsonSafely(command));

28
spring-data-mongodb/src/main/java/org/springframework/data/mongodb/core/ReactiveMongoTemplate.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2016-2017 the original author or authors.
* Copyright 2016-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -125,6 +125,7 @@ import com.mongodb.MongoException;
import com.mongodb.ReadPreference;
import com.mongodb.WriteConcern;
import com.mongodb.client.model.CreateCollectionOptions;
import com.mongodb.client.model.DeleteOptions;
import com.mongodb.client.model.Filters;
import com.mongodb.client.model.FindOneAndDeleteOptions;
import com.mongodb.client.model.FindOneAndUpdateOptions;
@@ -1611,27 +1612,34 @@ public class ReactiveMongoTemplate implements ReactiveMongoOperations, Applicati
return execute(collectionName, collection -> {
maybeEmitEvent(new BeforeDeleteEvent<T>(queryObject, entityClass, collectionName));
Document removeQuey = queryMapper.getMappedObject(queryObject, entity);
Document dboq = queryMapper.getMappedObject(queryObject, entity);
maybeEmitEvent(new BeforeDeleteEvent<T>(removeQuey, entityClass, collectionName));
MongoAction mongoAction = new MongoAction(writeConcern, MongoActionOperation.REMOVE, collectionName, entityClass,
null, queryObject);
null, removeQuey);
final DeleteOptions deleteOptions = new DeleteOptions();
query.getCollation().map(Collation::toMongoCollation).ifPresent(deleteOptions::collation);
WriteConcern writeConcernToUse = prepareWriteConcern(mongoAction);
MongoCollection<Document> collectionToUse = prepareCollection(collection, writeConcernToUse);
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Remove using query: {} in collection: {}.",
new Object[] { serializeToJsonSafely(dboq), collectionName });
new Object[] { serializeToJsonSafely(removeQuey), collectionName });
}
query.getCollation().ifPresent(val -> {
if (query.getLimit() > 0 || query.getSkip() > 0) {
// TODO: add collation support as soon as it's there! See https://jira.mongodb.org/browse/JAVARS-27
throw new IllegalArgumentException("DeleteMany does currently not accept collation settings.");
FindPublisher<Document> cursor = new QueryFindPublisherPreparer(query, entityClass)
.prepare(collection.find(removeQuey)).projection(new Document(ID_FIELD, 1));
return Flux.from(cursor).map(doc -> doc.get(ID_FIELD)).collectList().flatMap(val -> {
return Mono.from(collectionToUse.deleteMany(new Document(ID_FIELD, new Document("$in", val)), deleteOptions));
});
return collectionToUse.deleteMany(dboq);
} else {
return collectionToUse.deleteMany(removeQuey, deleteOptions);
}
}).doOnNext(deleteResult -> maybeEmitEvent(new AfterDeleteEvent<T>(queryObject, entityClass, collectionName)))
.next();

44
spring-data-mongodb/src/test/java/org/springframework/data/mongodb/core/MongoTemplateTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2011-2017 the original author or authors.
* Copyright 2011-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -33,18 +33,7 @@ import lombok.NoArgsConstructor;
import java.lang.reflect.InvocationTargetException;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Calendar;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.UUID;
import java.util.*;
import org.bson.types.ObjectId;
import org.hamcrest.collection.IsMapContaining;
@@ -108,6 +97,7 @@ import com.mongodb.client.FindIterable;
import com.mongodb.client.ListIndexesIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.result.DeleteResult;
import com.mongodb.client.result.UpdateResult;
/**
@@ -3286,6 +3276,34 @@ public class MongoTemplateTests {
assertThat(template.find(new Query().limit(1), Sample.class)).hasSize(1);
}
@Test // DATAMONGO-1870
public void removeShouldConsiderLimit() {
for (int i = 0; i < 100; i++) {
template.save(new Sample("id-" + i, i % 2 == 0 ? "stark" : "lannister"));
}
DeleteResult wr = template.remove(query(where("field").is("lannister")).limit(25), Sample.class);
assertThat(wr.getDeletedCount()).isEqualTo(25L);
assertThat(template.count(new Query(), Sample.class)).isEqualTo(75L);
}
@Test // DATAMONGO-1870
public void removeShouldConsiderSkipAndSort() {
for (int i = 0; i < 100; i++) {
template.save(new Sample("id-" + i, i % 2 == 0 ? "stark" : "lannister"));
}
DeleteResult wr = template.remove(new Query().skip(25).with(Sort.by("field")), Sample.class);
assertThat(wr.getDeletedCount()).isEqualTo(75L);
assertThat(template.count(new Query(), Sample.class)).isEqualTo(25L);
assertThat(template.count(query(where("field").is("lannister")), Sample.class)).isEqualTo(25L);
assertThat(template.count(query(where("field").is("stark")), Sample.class)).isEqualTo(0L);
}
static class TypeWithNumbers {
@Id String id;

5
spring-data-mongodb/src/test/java/org/springframework/data/mongodb/core/MongoTemplateUnitTests.java
View File

@@ -58,7 +58,6 @@ import org.springframework.data.domain.Sort;
import org.springframework.data.geo.Point;
import org.springframework.data.mongodb.MongoDbFactory;
import org.springframework.data.mongodb.core.aggregation.Aggregation;
import org.springframework.data.mongodb.core.query.Collation;
import org.springframework.data.mongodb.core.convert.DefaultDbRefResolver;
import org.springframework.data.mongodb.core.convert.MappingMongoConverter;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;
@@ -71,6 +70,7 @@ import org.springframework.data.mongodb.core.mapping.event.BeforeConvertEvent;
import org.springframework.data.mongodb.core.mapreduce.GroupBy;
import org.springframework.data.mongodb.core.mapreduce.MapReduceOptions;
import org.springframework.data.mongodb.core.query.BasicQuery;
import org.springframework.data.mongodb.core.query.Collation;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.NearQuery;
import org.springframework.data.mongodb.core.query.Query;
@@ -155,7 +155,7 @@ public class MongoTemplateUnitTests extends MongoOperationsUnitTests {
new MongoTemplate(null, "database");
}
@Test(expected = DataAccessException.class)
@Test(expected = IllegalArgumentException.class) // DATAMONGO-1870
public void removeHandlesMongoExceptionProperly() throws Exception {
MongoTemplate template = mockOutGetDb();
@@ -484,7 +484,6 @@ public class MongoTemplateUnitTests extends MongoOperationsUnitTests {
when(output.iterator()).thenReturn(cursor);
when(cursor.hasNext()).thenReturn(false);
when(collection.mapReduce(anyString(), anyString())).thenReturn(output);
template.mapReduce("collection", "function(){}", "function(key,values){}", new MapReduceOptions().limit(1000),

43
spring-data-mongodb/src/test/java/org/springframework/data/mongodb/core/ReactiveMongoTemplateTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2016-2017 the original author or authors.
* Copyright 2016-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -35,6 +35,7 @@ import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.assertj.core.api.Assertions;
import org.bson.Document;
import org.bson.types.ObjectId;
import org.junit.After;
@@ -298,9 +299,12 @@ public class ReactiveMongoTemplateTests {
public void updateFirstByEntityTypeShouldUpdateObject() {
Person person = new Person("Oliver2", 25);
StepVerifier.create(template.insert(person) //
.then(template.updateFirst(new Query(where("age").is(25)), new Update().set("firstName", "Sven"), Person.class)) //
.flatMapMany(p -> template.find(new Query(where("age").is(25)), Person.class))).consumeNextWith(actual -> {
StepVerifier
.create(template.insert(person) //
.then(template.updateFirst(new Query(where("age").is(25)), new Update().set("firstName", "Sven"),
Person.class)) //
.flatMapMany(p -> template.find(new Query(where("age").is(25)), Person.class)))
.consumeNextWith(actual -> {
assertThat(actual.getFirstName(), is(equalTo("Sven")));
}).verifyComplete();
@@ -481,7 +485,7 @@ public class ReactiveMongoTemplateTests {
@Test(expected = IllegalArgumentException.class) // DATAMONGO-1774
public void removeWithNullShouldThrowError() {
template.remove((Object)null).subscribe();
template.remove((Object) null).subscribe();
}
@Test // DATAMONGO-1774
@@ -924,6 +928,35 @@ public class ReactiveMongoTemplateTests {
assertThat(documents.poll(1, TimeUnit.SECONDS), is(nullValue()));
}
@Test // DATAMONGO-1870
public void removeShouldConsiderLimit() {
for (int i = 0; i < 100; i++) {
StepVerifier.create(template.save(new Sample("id-" + i, i % 2 == 0 ? "stark" : "lannister"))).expectNextCount(1)
.verifyComplete();
}
StepVerifier.create(template.remove(query(where("field").is("lannister")).limit(25), Sample.class))
.assertNext(wr -> Assertions.assertThat(wr.getDeletedCount()).isEqualTo(25L)).verifyComplete();
}
@Test // DATAMONGO-1870
public void removeShouldConsiderSkipAndSort() {
for (int i = 0; i < 100; i++) {
StepVerifier.create(template.save(new Sample("id-" + i, i % 2 == 0 ? "stark" : "lannister"))).expectNextCount(1)
.verifyComplete();
}
StepVerifier.create(template.remove(new Query().skip(25).with(Sort.by("field")), Sample.class))
.assertNext(wr -> Assertions.assertThat(wr.getDeletedCount()).isEqualTo(75L)).verifyComplete();
StepVerifier.create(template.count(query(where("field").is("lannister")), Sample.class)).expectNext(25L)
.verifyComplete();
StepVerifier.create(template.count(query(where("field").is("stark")), Sample.class)).expectNext(0L)
.verifyComplete();
}
private PersonWithAList createPersonWithAList(String firstname, int age) {
PersonWithAList p = new PersonWithAList();

8
spring-data-mongodb/src/test/java/org/springframework/data/mongodb/core/ReactiveMongoTemplateUnitTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2016-2017 the original author or authors.
* Copyright 2016-2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -168,16 +168,16 @@ public class ReactiveMongoTemplateUnitTests {
assertThat(options.getValue().getCollation().getLocale(), is("fr"));
}
@Ignore("see https://jira.mongodb.org/browse/JAVARS-27")
@Test // DATAMONGO-1518
public void findAndRemoveManyShouldUseCollationWhenPresent() {
when(collection.deleteMany(any(), any())).thenReturn(Mono.empty());
template.doRemove("collection-1", new BasicQuery("{}").collation(Collation.of("fr")), AutogenerateableId.class)
.subscribe();
ArgumentCaptor<DeleteOptions> options = ArgumentCaptor.forClass(DeleteOptions.class);
// the current mongodb-driver-reactivestreams:1.4.0 driver does not offer deleteMany with options.
// verify(collection).deleteMany(Mockito.any(), options.capture());
verify(collection).deleteMany(Mockito.any(), options.capture());
assertThat(options.getValue().getCollation().getLocale(), is("fr"));
}

20
src/main/asciidoc/reference/mongodb.adoc
View File

@@ -951,7 +951,25 @@ assertThat(p.getAge(), is(1));
You can use several overloaded methods to remove an object from the database.
* *remove* Remove the given document based on one of the following: a specific object instance, a query document criteria combined with a class or a query document criteria combined with a specific collection name.
====
[source,java]
----
template.remove(tywin, "GOT"); <1>
template.remove(query(where("lastname").is("lannister")), "GOT"); <2>
template.remove(new Query().limit(3), "GOT"); <3>
template.findAllAndRemove(query(where("lastname").is("lannister"), "GOT"); <4>
template.findAllAndRemove(new Query().limit(3), "GOT"); <5>
----
<1> Remove a single entity via its `id` from the associated collection.
<2> Remove all documents matching the criteria of the query from the `GOT` collection.
<3> Rewmove the first 3 documents in the `GOT` collection. Unlike <2> the documents to remove are identified via their `id` using the given query applying `sort`, `limit` and `skip` options and then removed all at once in a seperate step.
<4> Remove all documents matching the criteria of the query from the `GOT` collection. Unlike <3> documents do not get deleted in a batch but one by one.
<5> Remove the first 3 documents in the `GOT` collection. Unlike <3> documents do not get deleted in a batch but one by one.
====
[[mongo-template.optimistic-locking]]
=== Optimistic locking